{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/integrations/kumo/personalized_movie_search.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/integrations/kumo/personalized_movie_search.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "70hdOk9Gs_zU"
      },
      "source": [
        "\n",
        "<img src=\"https://kumo-ai.github.io/kumo-sdk/docs/_static/kumo-logo.svg\" width=auto height=\"100\">  <img src=\"https://kumo-sdk-public.s3.us-west-2.amazonaws.com/Pinecone-Primary-Logo-Black.png\" width=auto height=100>\n",
        "\n",
        "\n",
        "**This notebook requires a Kumo API key. To provision one for free and get started, visit https://kumo.ai/try/**.\n",
        "\n",
        "\n",
        "Your API key and environment will be emailed to you shortly after submitting the form on the website.\n",
        "\n",
        "---\n",
        "\n",
        "This notebook demonstrates and end-to-end example of building personalization products with Kumo and Pinecone. We will be working with a sampled [MovieLens25M dataset](https://grouplens.org/datasets/movielens/25m/). The dataset contains `users`, `movies`, and their `ratings`. We will use the data to first build a [relational deep learning](https://arxiv.org/abs/2312.04615) model which will output both *user* and *movie* embeddings. After the model is built and we've produced our embeddings, we will store them in [Pinecone](https://www.pinecone.io/) Serverless index(es), and explore how we can quickly build personalized products for our users.\n",
        "\n",
        "The recipe we give in this notebook is generalizable to many relevant personalization scenarios. The model training in Kumo is general, smooth, performant and scalable. The integration with Pinecone is seamless, and allows for rapid development of solutions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fBeinUZCnfJT"
      },
      "source": [
        "# Part 1: Building a (`user`, `movie`) embedding model in Kumo\n",
        "\n",
        "In this first part of the notebook we will train our very own (`user`, `movie`) embedding model.\n",
        "\n",
        "\n",
        "This is easy to do in Kumo, requires almost no data preprocessing. Kumo does heavy lifting, we only need to connect to the data, define the tables, graph, and the problem definition (predictive query). We can then use `kumo` to generate the training table, train the model, generate the prediction table, and finally output the embeddings.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eglM7b_AmG5Z"
      },
      "source": [
        "## Initialize Kumo\n",
        "\n",
        "Initializing the SDK is simple: install with `pip`, import, and connect to your Kumo platform endpoint using a provisioned API key.\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.init.html#kumoai.init).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "collapsed": true,
        "id": "-dKs0f2EaEcq",
        "outputId": "b22bdc96-bfc2-46bd-bf23-53f81d467842"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting kumoai\n",
            "  Downloading kumoai-1.2.4-py3-none-any.whl.metadata (2.1 kB)\n",
            "Collecting pandas (from kumoai)\n",
            "  Downloading pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (89 kB)\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m89.9/89.9 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting requests>=2.28.2 (from kumoai)\n",
            "  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)\n",
            "Collecting urllib3 (from kumoai)\n",
            "  Downloading urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)\n",
            "Collecting plotly (from kumoai)\n",
            "  Downloading plotly-6.0.1-py3-none-any.whl.metadata (6.7 kB)\n",
            "Collecting typing-extensions>=4.5.0 (from kumoai)\n",
            "  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)\n",
            "Collecting kumo-api==0.6.0 (from kumoai)\n",
            "  Downloading kumo_api-0.6.0-py3-none-any.whl.metadata (1.3 kB)\n",
            "Collecting tqdm>=4.66.0 (from kumoai)\n",
            "  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m57.7/57.7 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiohttp>=3.10.0 (from kumoai)\n",
            "  Downloading aiohttp-3.11.14-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)\n",
            "Collecting pydantic==1.10.13 (from kumo-api==0.6.0->kumoai)\n",
            "  Downloading pydantic-1.10.13-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (149 kB)\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m149.6/149.6 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiohappyeyeballs>=2.3.0 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading aiohappyeyeballs-2.6.1-py3-none-any.whl.metadata (5.9 kB)\n",
            "Collecting aiosignal>=1.1.2 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading aiosignal-1.3.2-py2.py3-none-any.whl.metadata (3.8 kB)\n",
            "Collecting attrs>=17.3.0 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading attrs-25.3.0-py3-none-any.whl.metadata (10 kB)\n",
            "Collecting frozenlist>=1.1.1 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading frozenlist-1.5.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)\n",
            "Collecting multidict<7.0,>=4.5 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading multidict-6.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n",
            "Collecting propcache>=0.2.0 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading propcache-0.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)\n",
            "Collecting yarl<2.0,>=1.17.0 (from aiohttp>=3.10.0->kumoai)\n",
            "  Downloading yarl-1.18.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (69 kB)\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m69.2/69.2 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting charset-normalizer<4,>=2 (from requests>=2.28.2->kumoai)\n",
            "  Downloading charset_normalizer-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (35 kB)\n",
            "Collecting idna<4,>=2.5 (from requests>=2.28.2->kumoai)\n",
            "  Downloading idna-3.10-py3-none-any.whl.metadata (10 kB)\n",
            "Collecting certifi>=2017.4.17 (from requests>=2.28.2->kumoai)\n",
            "  Downloading certifi-2025.1.31-py3-none-any.whl.metadata (2.5 kB)\n",
            "Collecting numpy>=1.23.2 (from pandas->kumoai)\n",
            "  Downloading numpy-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m62.0/62.0 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting python-dateutil>=2.8.2 (from pandas->kumoai)\n",
            "  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)\n",
            "Collecting pytz>=2020.1 (from pandas->kumoai)\n",
            "  Downloading pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)\n",
            "Collecting tzdata>=2022.7 (from pandas->kumoai)\n",
            "  Downloading tzdata-2025.2-py2.py3-none-any.whl.metadata (1.4 kB)\n",
            "Collecting narwhals>=1.15.1 (from plotly->kumoai)\n",
            "  Downloading narwhals-1.32.0-py3-none-any.whl.metadata (9.2 kB)\n",
            "Collecting packaging (from plotly->kumoai)\n",
            "  Downloading packaging-24.2-py3-none-any.whl.metadata (3.2 kB)\n",
            "Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas->kumoai)\n",
            "  Downloading six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)\n",
            "Downloading kumoai-1.2.4-py3-none-any.whl (98 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m98.6/98.6 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading kumo_api-0.6.0-py3-none-any.whl (37 kB)\n",
            "Downloading pydantic-1.10.13-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m58.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading aiohttp-3.11.14-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m27.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading requests-2.32.3-py3-none-any.whl (64 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m64.9/64.9 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tqdm-4.67.1-py3-none-any.whl (78 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)\n",
            "Downloading urllib3-2.3.0-py3-none-any.whl (128 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m128.4/128.4 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.1 MB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m13.1/13.1 MB\u001b[0m \u001b[31m45.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading plotly-6.0.1-py3-none-any.whl (14.8 MB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m14.8/14.8 MB\u001b[0m \u001b[31m43.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading aiohappyeyeballs-2.6.1-py3-none-any.whl (15 kB)\n",
            "Downloading aiosignal-1.3.2-py2.py3-none-any.whl (7.6 kB)\n",
            "Downloading attrs-25.3.0-py3-none-any.whl (63 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m63.8/63.8 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading certifi-2025.1.31-py3-none-any.whl (166 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m166.4/166.4 kB\u001b[0m \u001b[31m14.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading charset_normalizer-3.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (143 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m143.9/143.9 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading frozenlist-1.5.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (274 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m274.9/274.9 kB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading idna-3.10-py3-none-any.whl (70 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m70.4/70.4 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading multidict-6.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (133 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m133.3/133.3 kB\u001b[0m \u001b[31m11.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading narwhals-1.32.0-py3-none-any.whl (320 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m320.1/320.1 kB\u001b[0m \u001b[31m23.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading numpy-2.2.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.4 MB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m16.4/16.4 MB\u001b[0m \u001b[31m29.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading propcache-0.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (231 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m231.3/231.3 kB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m229.9/229.9 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pytz-2025.2-py2.py3-none-any.whl (509 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m509.2/509.2 kB\u001b[0m \u001b[31m25.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tzdata-2025.2-py2.py3-none-any.whl (347 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m347.8/347.8 kB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading yarl-1.18.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (344 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m344.1/344.1 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading packaging-24.2-py3-none-any.whl (65 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m65.5/65.5 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading six-1.17.0-py2.py3-none-any.whl (11 kB)\n",
            "Installing collected packages: pytz, urllib3, tzdata, typing-extensions, tqdm, six, propcache, packaging, numpy, narwhals, multidict, idna, frozenlist, charset-normalizer, certifi, attrs, aiohappyeyeballs, yarl, requests, python-dateutil, pydantic, plotly, aiosignal, pandas, kumo-api, aiohttp, kumoai\n",
            "  Attempting uninstall: pytz\n",
            "    Found existing installation: pytz 2025.1\n",
            "    Uninstalling pytz-2025.1:\n",
            "      Successfully uninstalled pytz-2025.1\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 2.3.0\n",
            "    Uninstalling urllib3-2.3.0:\n",
            "      Successfully uninstalled urllib3-2.3.0\n",
            "  Attempting uninstall: tzdata\n",
            "    Found existing installation: tzdata 2025.1\n",
            "    Uninstalling tzdata-2025.1:\n",
            "      Successfully uninstalled tzdata-2025.1\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing_extensions 4.12.2\n",
            "    Uninstalling typing_extensions-4.12.2:\n",
            "      Successfully uninstalled typing_extensions-4.12.2\n",
            "  Attempting uninstall: tqdm\n",
            "    Found existing installation: tqdm 4.67.1\n",
            "    Uninstalling tqdm-4.67.1:\n",
            "      Successfully uninstalled tqdm-4.67.1\n",
            "  Attempting uninstall: six\n",
            "    Found existing installation: six 1.17.0\n",
            "    Uninstalling six-1.17.0:\n",
            "      Successfully uninstalled six-1.17.0\n",
            "  Attempting uninstall: propcache\n",
            "    Found existing installation: propcache 0.3.0\n",
            "    Uninstalling propcache-0.3.0:\n",
            "      Successfully uninstalled propcache-0.3.0\n",
            "  Attempting uninstall: packaging\n",
            "    Found existing installation: packaging 24.2\n",
            "    Uninstalling packaging-24.2:\n",
            "      Successfully uninstalled packaging-24.2\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 2.0.2\n",
            "    Uninstalling numpy-2.0.2:\n",
            "      Successfully uninstalled numpy-2.0.2\n",
            "  Attempting uninstall: narwhals\n",
            "    Found existing installation: narwhals 1.31.0\n",
            "    Uninstalling narwhals-1.31.0:\n",
            "      Successfully uninstalled narwhals-1.31.0\n",
            "  Attempting uninstall: multidict\n",
            "    Found existing installation: multidict 6.2.0\n",
            "    Uninstalling multidict-6.2.0:\n",
            "      Successfully uninstalled multidict-6.2.0\n",
            "  Attempting uninstall: idna\n",
            "    Found existing installation: idna 3.10\n",
            "    Uninstalling idna-3.10:\n",
            "      Successfully uninstalled idna-3.10\n",
            "  Attempting uninstall: frozenlist\n",
            "    Found existing installation: frozenlist 1.5.0\n",
            "    Uninstalling frozenlist-1.5.0:\n",
            "      Successfully uninstalled frozenlist-1.5.0\n",
            "  Attempting uninstall: charset-normalizer\n",
            "    Found existing installation: charset-normalizer 3.4.1\n",
            "    Uninstalling charset-normalizer-3.4.1:\n",
            "      Successfully uninstalled charset-normalizer-3.4.1\n",
            "  Attempting uninstall: certifi\n",
            "    Found existing installation: certifi 2025.1.31\n",
            "    Uninstalling certifi-2025.1.31:\n",
            "      Successfully uninstalled certifi-2025.1.31\n",
            "  Attempting uninstall: attrs\n",
            "    Found existing installation: attrs 25.3.0\n",
            "    Uninstalling attrs-25.3.0:\n",
            "      Successfully uninstalled attrs-25.3.0\n",
            "  Attempting uninstall: aiohappyeyeballs\n",
            "    Found existing installation: aiohappyeyeballs 2.6.1\n",
            "    Uninstalling aiohappyeyeballs-2.6.1:\n",
            "      Successfully uninstalled aiohappyeyeballs-2.6.1\n",
            "  Attempting uninstall: yarl\n",
            "    Found existing installation: yarl 1.18.3\n",
            "    Uninstalling yarl-1.18.3:\n",
            "      Successfully uninstalled yarl-1.18.3\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.32.3\n",
            "    Uninstalling requests-2.32.3:\n",
            "      Successfully uninstalled requests-2.32.3\n",
            "  Attempting uninstall: python-dateutil\n",
            "    Found existing installation: python-dateutil 2.8.2\n",
            "    Uninstalling python-dateutil-2.8.2:\n",
            "      Successfully uninstalled python-dateutil-2.8.2\n",
            "  Attempting uninstall: pydantic\n",
            "    Found existing installation: pydantic 2.10.6\n",
            "    Uninstalling pydantic-2.10.6:\n",
            "      Successfully uninstalled pydantic-2.10.6\n",
            "  Attempting uninstall: plotly\n",
            "    Found existing installation: plotly 5.24.1\n",
            "    Uninstalling plotly-5.24.1:\n",
            "      Successfully uninstalled plotly-5.24.1\n",
            "  Attempting uninstall: aiosignal\n",
            "    Found existing installation: aiosignal 1.3.2\n",
            "    Uninstalling aiosignal-1.3.2:\n",
            "      Successfully uninstalled aiosignal-1.3.2\n",
            "  Attempting uninstall: pandas\n",
            "    Found existing installation: pandas 2.2.2\n",
            "    Uninstalling pandas-2.2.2:\n",
            "      Successfully uninstalled pandas-2.2.2\n",
            "  Attempting uninstall: aiohttp\n",
            "    Found existing installation: aiohttp 3.11.14\n",
            "    Uninstalling aiohttp-3.11.14:\n",
            "      Successfully uninstalled aiohttp-3.11.14\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "google-colab 1.0.0 requires pandas==2.2.2, but you have pandas 2.2.3 which is incompatible.\n",
            "wandb 0.19.8 requires pydantic<3,>=2.6, but you have pydantic 1.10.13 which is incompatible.\n",
            "numba 0.60.0 requires numpy<2.1,>=1.22, but you have numpy 2.2.4 which is incompatible.\n",
            "langchain-core 0.3.47 requires pydantic<3.0.0,>=2.5.2; python_full_version < \"3.12.4\", but you have pydantic 1.10.13 which is incompatible.\n",
            "albumentations 2.0.5 requires pydantic>=2.9.2, but you have pydantic 1.10.13 which is incompatible.\n",
            "tensorflow 2.18.0 requires numpy<2.1.0,>=1.26.0, but you have numpy 2.2.4 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cublas-cu12==12.4.5.8; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cublas-cu12 12.5.3.2 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cuda-cupti-cu12==12.4.127; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cuda-cupti-cu12 12.5.82 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cuda-nvrtc-cu12==12.4.127; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cuda-nvrtc-cu12 12.5.82 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cuda-runtime-cu12==12.4.127; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cuda-runtime-cu12 12.5.82 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cudnn-cu12==9.1.0.70; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cudnn-cu12 9.3.0.75 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cufft-cu12==11.2.1.3; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cufft-cu12 11.2.3.61 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-curand-cu12==10.3.5.147; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-curand-cu12 10.3.6.82 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cusolver-cu12==11.6.1.9; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cusolver-cu12 11.6.3.83 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-cusparse-cu12==12.3.1.170; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-cusparse-cu12 12.5.1.3 which is incompatible.\n",
            "torch 2.6.0+cu124 requires nvidia-nvjitlink-cu12==12.4.127; platform_system == \"Linux\" and platform_machine == \"x86_64\", but you have nvidia-nvjitlink-cu12 12.5.82 which is incompatible.\n",
            "langchain 0.3.21 requires pydantic<3.0.0,>=2.7.4, but you have pydantic 1.10.13 which is incompatible.\n",
            "google-genai 1.7.0 requires pydantic<3.0.0,>=2.0.0, but you have pydantic 1.10.13 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed aiohappyeyeballs-2.6.1 aiohttp-3.11.14 aiosignal-1.3.2 attrs-25.3.0 certifi-2025.1.31 charset-normalizer-3.4.1 frozenlist-1.5.0 idna-3.10 kumo-api-0.6.0 kumoai-1.2.4 multidict-6.2.0 narwhals-1.32.0 numpy-2.2.4 packaging-24.2 pandas-2.2.3 plotly-6.0.1 propcache-0.3.0 pydantic-1.10.13 python-dateutil-2.9.0.post0 pytz-2025.2 requests-2.32.3 six-1.17.0 tqdm-4.67.1 typing-extensions-4.12.2 tzdata-2025.2 urllib3-2.3.0 yarl-1.18.3\n"
          ]
        },
        {
          "data": {
            "application/vnd.colab-display-data+json": {
              "id": "370173b2b82c4336976176214efd698f",
              "pip_warning": {
                "packages": [
                  "certifi",
                  "dateutil",
                  "pytz",
                  "six"
                ]
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "!pip install kumoai --force-reinstall"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KfgEsvLKcwMG"
      },
      "outputs": [],
      "source": [
        "API_KEY = '<your-api-key>'\n",
        "ENVIRONMENT = '<your-provisioned-environment>'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GFViCuHzc2ev",
        "outputId": "ca753ec7-2540-4599-f2d5-89b4eb0dd846"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:kumoai:Successfully initialized the Kumo SDK against deployment https://demo-sdk.kumoai.cloud/api, with log level INFO.\n"
          ]
        }
      ],
      "source": [
        "import kumoai as kumo\n",
        "from kumoai.graph import Column, Table, Edge, Graph\n",
        "from kumoai.connector import S3Connector\n",
        "from kumoai.pquery import RunMode\n",
        "from kumoai.trainer.config import OutputConfig\n",
        "\n",
        "kumo.init(url=ENVIRONMENT, api_key=API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tp99QELtY9hf"
      },
      "source": [
        "## Connecting Data\n",
        "\n",
        "You can connect data to the Kumo platform from a variety of data sources: see [`kumo.Connector`](https://kumo-ai.github.io/kumo-sdk/docs/modules/connector.html) for more details. We support connecting to data on Snowflake, Databricks, BigQuery, and Amazon S3.\n",
        "\n",
        "For the purposes of this notebook example we stored the dataset in `s3://kumo-public-datasets/movielens/basic_with_posters/`. The data is sampled from [MovieLens25](https://grouplens.org/datasets/movielens/25m/) and contains three tables `users`, `movies`, and `ratings`. To learn more about the data visit:\n",
        "1. [pinecone/movie-posters](https://huggingface.co/datasets/pinecone/movie-posters)\n",
        "2. [pinecone/movielens-recent-ratings](https://huggingface.co/datasets/pinecone/movielens-recent-ratings)\n",
        "\n",
        "\n",
        "*To learn more about connecting to your data source see documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/modules/connector.html)*\n",
        "\n",
        "<img src=\"https://kumo-sdk-public.s3.us-west-2.amazonaws.com/kumo_data.png\" alt=\"drawing\" width=\"800\"/>\n",
        "y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KNynmZ7ia6BZ"
      },
      "outputs": [],
      "source": [
        "connector = kumo.S3Connector(root_dir='s3://kumo-demo-datasets/movielens/basic_with_posters/')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qQK7WZ9MvxI"
      },
      "source": [
        "Connectors can be used to inspect the tables within them, and fetch samples of the source data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bzm-b-TSCQgA",
        "outputId": "ac6899e5-b317-42a3-8af2-8e80728400e4"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['movies', 'ratings', 'users']"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# List all table names behind this connector:\n",
        "connector.table_names()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "ZdQ3A0Q24gYa",
        "outputId": "0c9e9af2-6f68-41f9-c1ca-e3ec6b243fd0"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"connector['users']\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"userId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"49821\",\n          \"59617\",\n          \"55491\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-af96f510-fc3b-4c42-9be6-4bc9ca38018b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6252</th>\n",
              "      <td>51926</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4684</th>\n",
              "      <td>49821</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1731</th>\n",
              "      <td>55491</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4742</th>\n",
              "      <td>63093</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4521</th>\n",
              "      <td>59617</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-af96f510-fc3b-4c42-9be6-4bc9ca38018b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-af96f510-fc3b-4c42-9be6-4bc9ca38018b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-af96f510-fc3b-4c42-9be6-4bc9ca38018b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-95baf39d-c33d-43b3-ba68-d859fe288205\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-95baf39d-c33d-43b3-ba68-d859fe288205')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-95baf39d-c33d-43b3-ba68-d859fe288205 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "     userId\n",
              "6252  51926\n",
              "4684  49821\n",
              "1731  55491\n",
              "4742  63093\n",
              "4521  59617"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# View a sample of the 'users' table's rows:\n",
        "connector['users'].head(num_rows=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "gGXrM3j45LWU",
        "outputId": "837a042e-c818-43c2-ef71-cd175725a672"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"connector['movies']\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"imdbId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"2586118\",\n          \"3894312\",\n          \"3591984\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"movieId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"163474\",\n          \"162162\",\n          \"131808\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"title\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"The Ardennes (2015)\",\n          \"The Remains (2016)\",\n          \"The Postman's White Nights (2014)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"genres\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"Crime|Drama\",\n          \"Horror\",\n          \"(no genres listed)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tmdbId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"351037\",\n          \"407389\",\n          \"283703\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"poster\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"https://m.media-amazon.com/images/M/MV5BNTA4ODQwOTItODAwOS00YmQ1LTg4OGMtZjRlN2YyNTM1YzU4XkEyXkFqcGdeQXVyMjA0MzYwMDY@._V1_SX300.jpg\",\n          \"https://m.media-amazon.com/images/M/MV5BMjA2MjE1MzA4NV5BMl5BanBnXkFtZTgwNzUxMzc1OTE@._V1_SX300.jpg\",\n          \"https://m.media-amazon.com/images/M/MV5BMTkxMTAzNTQ5M15BMl5BanBnXkFtZTgwODMyOTU2MjE@._V1_SX300.jpg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-c1a1c557-f8b6-4919-b605-c3ca18ebeccc\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>imdbId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>tmdbId</th>\n",
              "      <th>poster</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6252</th>\n",
              "      <td>4289434</td>\n",
              "      <td>174479</td>\n",
              "      <td>Unedited Footage of a Bear (2014)</td>\n",
              "      <td>Horror|Thriller</td>\n",
              "      <td>312174</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BYzIzOG...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4684</th>\n",
              "      <td>2586118</td>\n",
              "      <td>163474</td>\n",
              "      <td>The Ardennes (2015)</td>\n",
              "      <td>Crime|Drama</td>\n",
              "      <td>351037</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BNTA4OD...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1731</th>\n",
              "      <td>3591984</td>\n",
              "      <td>131808</td>\n",
              "      <td>The Postman's White Nights (2014)</td>\n",
              "      <td>(no genres listed)</td>\n",
              "      <td>283703</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMTkxMT...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4742</th>\n",
              "      <td>4730986</td>\n",
              "      <td>163909</td>\n",
              "      <td>Divines (2016)</td>\n",
              "      <td>Drama</td>\n",
              "      <td>393729</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BYTE2ND...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4521</th>\n",
              "      <td>3894312</td>\n",
              "      <td>162162</td>\n",
              "      <td>The Remains (2016)</td>\n",
              "      <td>Horror</td>\n",
              "      <td>407389</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMjA2Mj...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c1a1c557-f8b6-4919-b605-c3ca18ebeccc')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c1a1c557-f8b6-4919-b605-c3ca18ebeccc button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c1a1c557-f8b6-4919-b605-c3ca18ebeccc');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-1fc11453-f454-4ec7-a9cb-e837ec7dd3bb\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-1fc11453-f454-4ec7-a9cb-e837ec7dd3bb')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-1fc11453-f454-4ec7-a9cb-e837ec7dd3bb button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       imdbId movieId                              title              genres  \\\n",
              "6252  4289434  174479  Unedited Footage of a Bear (2014)     Horror|Thriller   \n",
              "4684  2586118  163474                The Ardennes (2015)         Crime|Drama   \n",
              "1731  3591984  131808  The Postman's White Nights (2014)  (no genres listed)   \n",
              "4742  4730986  163909                     Divines (2016)               Drama   \n",
              "4521  3894312  162162                 The Remains (2016)              Horror   \n",
              "\n",
              "      tmdbId                                             poster  \n",
              "6252  312174  https://m.media-amazon.com/images/M/MV5BYzIzOG...  \n",
              "4684  351037  https://m.media-amazon.com/images/M/MV5BNTA4OD...  \n",
              "1731  283703  https://m.media-amazon.com/images/M/MV5BMTkxMT...  \n",
              "4742  393729  https://m.media-amazon.com/images/M/MV5BYTE2ND...  \n",
              "4521  407389  https://m.media-amazon.com/images/M/MV5BMjA2Mj...  "
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# View a sample of the 'games' table's rows:\n",
        "connector['movies'].head(num_rows=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "Sw1XMOKBTG8T",
        "outputId": "ec81f340-e6f4-4bb5-acae-9a98f159fdf9"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"connector['ratings']\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"movieId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"135143\",\n          \"201586\",\n          \"164179\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"userId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"162055\",\n          \"161214\",\n          \"162271\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"rating\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"1.5\",\n          \"4.0\",\n          \"2.5\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"timestamp\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": \"2016-06-18 10:52:20\",\n        \"max\": \"2019-08-20 21:47:55\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"2017-03-26 13:45:45\",\n          \"2019-06-15 10:38:42\",\n          \"2017-03-26 13:19:41\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-8445fbf9-0a1d-4aeb-af40-aa9a91e2b698\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>userId</th>\n",
              "      <th>rating</th>\n",
              "      <th>timestamp</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6252</th>\n",
              "      <td>188215</td>\n",
              "      <td>162271</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2019-08-20 21:47:55</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4684</th>\n",
              "      <td>135143</td>\n",
              "      <td>162055</td>\n",
              "      <td>1.5</td>\n",
              "      <td>2017-03-26 13:45:45</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1731</th>\n",
              "      <td>164179</td>\n",
              "      <td>162055</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2017-03-26 13:19:41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4742</th>\n",
              "      <td>136562</td>\n",
              "      <td>161474</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016-06-18 10:52:20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4521</th>\n",
              "      <td>201586</td>\n",
              "      <td>161214</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2019-06-15 10:38:42</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8445fbf9-0a1d-4aeb-af40-aa9a91e2b698')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8445fbf9-0a1d-4aeb-af40-aa9a91e2b698 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8445fbf9-0a1d-4aeb-af40-aa9a91e2b698');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2cd81660-bd5a-415b-b459-a304784f96e0\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2cd81660-bd5a-415b-b459-a304784f96e0')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2cd81660-bd5a-415b-b459-a304784f96e0 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "     movieId  userId rating           timestamp\n",
              "6252  188215  162271    2.5 2019-08-20 21:47:55\n",
              "4684  135143  162055    1.5 2017-03-26 13:45:45\n",
              "1731  164179  162055    5.0 2017-03-26 13:19:41\n",
              "4742  136562  161474    4.0 2016-06-18 10:52:20\n",
              "4521  201586  161214    4.0 2019-06-15 10:38:42"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# View a sample of the 'game_sessions' table's rows:\n",
        "connector['ratings'].head(num_rows=5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w80jJDnaVmuo"
      },
      "source": [
        "## Creating Tables\n",
        "\n",
        "Once we are comfortable with our source data, we can prepare data for the Kumo platform by constructing Kumo `Table` objects from the source tables. Kumo `Table` objects define important metadata for the downstream machine learning problem, including\n",
        "* Column data types (`dtype`) and semantic types (`stype`)\n",
        "* The table's primary key, if present\n",
        "* The table's time and end time columns, if present\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.graph.Table.html).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Q6VQ943cItH"
      },
      "outputs": [],
      "source": [
        "# Create a Kumo table from a source table, specifying the primary_key\n",
        "users = Table.from_source_table(\n",
        "    source_table=connector['users'],\n",
        "    primary_key='userId',\n",
        ")\n",
        "\n",
        "# # For any table created from source we can print it's definition\n",
        "# users.print_definition()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-0mEkuRYmIyK"
      },
      "outputs": [],
      "source": [
        "movies = Table(\n",
        "  source_table=connector[\"movies\"],\n",
        "  primary_key=\"movieId\",\n",
        "  columns=[\n",
        "    Column(name=\"movieId\", stype=\"ID\", dtype=\"string\"),\n",
        "    Column(name=\"title\", stype=\"text\", dtype=\"string\"),\n",
        "    Column(name=\"genres\", stype=\"multicategorical\", dtype=\"string\"),\n",
        "  ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1qckKUrRmprC"
      },
      "outputs": [],
      "source": [
        "ratings = Table(\n",
        "  source_table=connector[\"ratings\"],\n",
        "  primary_key=None,\n",
        "  time_column=\"timestamp\",\n",
        "  end_time_column=None,\n",
        "  columns=[\n",
        "    Column(name=\"movieId\", stype=\"ID\", dtype=\"string\"),\n",
        "    Column(name=\"userId\", stype=\"ID\", dtype=\"string\"),\n",
        "    Column(name=\"rating\", stype=\"categorical\", dtype=\"float32\"),\n",
        "    Column(name=\"timestamp\", stype=\"timestamp\", dtype=\"time\"),\n",
        "  ],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BNu5DSWkneuc"
      },
      "source": [
        "## Creating a Graph\n",
        "\n",
        "After specifying our Kumo tables, we can next create a `Graph`, which represents relationships between these tables. Defining this graph is the final step of the data specification pipeline; after its creation, we are able to create predictive queries to answer business problems that relate to our data.\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.graph.Graph.html).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        },
        "id": "U5DwLqSpnrs6",
        "outputId": "73a43922-495e-4759-a10b-f3eb6856485b"
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.43.0 (0)\n",
              " -->\n",
              "<!-- Title: %3 Pages: 1 -->\n",
              "<svg width=\"198pt\" height=\"218pt\"\n",
              " viewBox=\"0.00 0.00 198.00 218.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 214)\">\n",
              "<title>%3</title>\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-214 194,-214 194,4 -4,4\"/>\n",
              "<!-- users -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>users</title>\n",
              "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-46.5 80,-46.5 80,-0.5 0,-0.5\"/>\n",
              "<text text-anchor=\"middle\" x=\"40\" y=\"-31.3\" font-family=\"Times,serif\" font-size=\"14.00\">users</text>\n",
              "<polyline fill=\"none\" stroke=\"black\" points=\"0,-23.5 80,-23.5 \"/>\n",
              "<text text-anchor=\"start\" x=\"8\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\">userId (PK)</text>\n",
              "</g>\n",
              "<!-- movies -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>movies</title>\n",
              "<polygon fill=\"none\" stroke=\"black\" points=\"98,-0.5 98,-46.5 190,-46.5 190,-0.5 98,-0.5\"/>\n",
              "<text text-anchor=\"middle\" x=\"144\" y=\"-31.3\" font-family=\"Times,serif\" font-size=\"14.00\">movies</text>\n",
              "<polyline fill=\"none\" stroke=\"black\" points=\"98,-23.5 190,-23.5 \"/>\n",
              "<text text-anchor=\"start\" x=\"106\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\">movieId (PK)</text>\n",
              "</g>\n",
              "<!-- ratings -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>ratings</title>\n",
              "<polygon fill=\"none\" stroke=\"black\" points=\"34,-133.5 34,-209.5 150,-209.5 150,-133.5 34,-133.5\"/>\n",
              "<text text-anchor=\"middle\" x=\"92\" y=\"-194.3\" font-family=\"Times,serif\" font-size=\"14.00\">ratings</text>\n",
              "<polyline fill=\"none\" stroke=\"black\" points=\"34,-186.5 150,-186.5 \"/>\n",
              "<text text-anchor=\"start\" x=\"42\" y=\"-171.3\" font-family=\"Times,serif\" font-size=\"14.00\">userId (FK)</text>\n",
              "<text text-anchor=\"start\" x=\"42\" y=\"-156.3\" font-family=\"Times,serif\" font-size=\"14.00\">movieId (FK)</text>\n",
              "<text text-anchor=\"start\" x=\"42\" y=\"-141.3\" font-family=\"Times,serif\" font-size=\"14.00\">timestamp (Time)</text>\n",
              "</g>\n",
              "<!-- ratings&#45;&#45;users -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>ratings&#45;&#45;users</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M78.74,-133.28C68.99,-105.91 56.07,-69.62 47.89,-46.66\"/>\n",
              "<text text-anchor=\"middle\" x=\"83\" y=\"-87.2\" font-family=\"Times,serif\" font-size=\"11.00\"> userId </text>\n",
              "<text text-anchor=\"middle\" x=\"58.42\" y=\"-54.54\" font-family=\"Times,serif\" font-size=\"11.00\">1</text>\n",
              "<text text-anchor=\"middle\" x=\"68.21\" y=\"-119.8\" font-family=\"Times,serif\" font-size=\"11.00\">*</text>\n",
              "</g>\n",
              "<!-- ratings&#45;&#45;movies -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>ratings&#45;&#45;movies</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M105.26,-133.28C115.01,-105.91 127.93,-69.62 136.11,-46.66\"/>\n",
              "<text text-anchor=\"middle\" x=\"143.5\" y=\"-87.2\" font-family=\"Times,serif\" font-size=\"11.00\"> movieId </text>\n",
              "<text text-anchor=\"middle\" x=\"137.52\" y=\"-58.79\" font-family=\"Times,serif\" font-size=\"11.00\">1</text>\n",
              "<text text-anchor=\"middle\" x=\"103.85\" y=\"-115.55\" font-family=\"Times,serif\" font-size=\"11.00\">*</text>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.graphs.Graph at 0x7d9ea7274c90>"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "graph = Graph(\n",
        "\t# These are the tables that participate in the graph: the keys of this\n",
        "\t# dictionary are the names of the tables, and the values are the Table\n",
        "\t# objects that correspond to these names:\n",
        "\ttables={\n",
        "    'users' : users,\n",
        "    'movies' : movies,\n",
        "    'ratings' : ratings\n",
        "\t},\n",
        "\n",
        " \t# These are the edges that define the primary key / foreign key\n",
        "\t# relationships between the tables defined above. Here, `src_table`\n",
        "\t# is the table that has the foreign key `fkey`, which maps to the\n",
        "\t# table `dst_table`'s primary key:`\n",
        "\tedges=[\n",
        "    Edge(src_table='ratings', dst_table='users', fkey='userId'),\n",
        "    Edge(src_table='ratings', dst_table='movies', fkey='movieId'),\n",
        "\t],\n",
        ")\n",
        "\n",
        "graph.visualize(show_cols=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gu3Qt2Uln8AQ"
      },
      "source": [
        "## Writing a Predictive Query\n",
        "\n",
        "After we've connected our data as Kumo Tables in a Kumo Graph, we can write a predictive query representing a business problem we would like to solve on our specified tables; please see the Kumo documentation for the specification of the predictive query language.\n",
        "\n",
        "<img src=\"https://kumo-sdk-public.s3.us-west-2.amazonaws.com/kumo_pq.png\" alt=\"drawing\" width=\"700\"/>\n",
        "\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.pquery.PredictiveQuery.html#kumoai.pquery.PredictiveQuery).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t4-oB33kuVJ0"
      },
      "source": [
        "### Aside: what to think about when thinking about PQs\n",
        "When writing your predictive query, always focus on the underlying business problem you\u2019re solving. In our case, we aim to build powerful personalization products, which require robust user and movie embeddings that we can easily search and compare.\n",
        "\n",
        "There are two key considerations when building an embedding model with Kumo:\n",
        "1.\t**Pick supervision signal:** Supervision guides the embedding model by defining positive and negative interactions between users and movies. You can flexibly define what constitutes a positive event based on your business goals, directly influencing how the embedding space captures meaningful user-movie relationships.\n",
        "  - Keep in mind that every positive (`user`, `movie`) interaction brings that pair closer in embedding space, and each negative interaction pushes them apart. After training we end up with both representations in the same embedding space.\n",
        "2.\t**Model Architecture:** Clearly specify your desired architecture when setting the `model_plan` parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fqj1vjlSoKI_",
        "outputId": "cccf4ee8-b3c5-452a-9f80-f0e2bff14654"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "INFO:kumoai.pquery.predictive_query:Query PREDICT LIST_DISTINCT(ratings.movieId, 0, 14, days) RANK TOP 25 FOR EACH users.userId is configured correctly.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<kumoai.pquery.predictive_query.PredictiveQuery at 0x7d9ea72b2b50>"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Construct a query to predict movies (movieId) the user is most likely to interact with\n",
        "query = kumo.PredictiveQuery(\n",
        "\tgraph=graph,\n",
        "\tquery=\"PREDICT LIST_DISTINCT(ratings.movieId, 0, 14, days) RANK TOP 25 FOR EACH users.userId\",\n",
        ")\n",
        "\n",
        "# Ensure this query is specified appropriately for this graph:\n",
        "query.validate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EqE_H2YRXqsN"
      },
      "source": [
        "### Aside: On predictive queries...\n",
        "Predictive queries are exactly what the name states, they are very much like *queries* which the reader might be familar with from SQL, but they are *predictive* i.e. the aggregations are *future* looking.  \n",
        "\n",
        "Of course, since predictive queries are *future* looking we need to train a machine learning model to actually execute or rather answer the query. But once the model is trained the predictive query becomes *executable*, almost as if the underlying tables included records from the future. Of course, under the hood this just amounts to model inference, but if the model is trained well the exemplum above is very close to the truth.\n",
        "\n",
        "#### Writing other queries\n",
        "The Kumo abstraction is powerful precisely because there is no need to modify the underlying data in order to answer different *predictive queries* - the graph representation and our models make this possible. Here are some other queries you can define on the same graph we defined above.\n",
        "\n",
        "A) **predicting user churn**\n",
        "```\n",
        "PREDICT COUNT(ratings.*, 0, 30, days) = 0\n",
        "FOR EACH users.userId\n",
        "```\n",
        "\n",
        "B) **predicting user rating sum**\n",
        "```\n",
        "PREDICT SUM(ratings.rating, 0, 30, days)\n",
        "FOR EACH users.userId\n",
        "```\n",
        "\n",
        "C) **forecast the count of users who will watch a particular movie**\n",
        "```\n",
        "PREDICT COUNT(ratings.userId, 0, 14, days)\n",
        "FOR EACH movie.movieId\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "syi-o-Rt85kd",
        "outputId": "b4e85e5b-ac63-44a1-b2c4-0d65143d3c06"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "This query is a temporal link prediction task.\n"
          ]
        }
      ],
      "source": [
        "# Fetch the machine learning task type for this query:\n",
        "print(f\"This query is a {query.get_task_type().replace('_', ' ')} task.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "107-I1Rm9Zzy"
      },
      "source": [
        "## Training a Model\n",
        "\n",
        "With a predictive query in place, we can now train a model to predict the desired outputs of the query over our Kumo Graph. The Kumo SDK supports modular execution of the different components of the training pipeline for ease of experimentation and hyperparameter tuning.\n",
        "\n",
        "### Generating a Training Table\n",
        "\n",
        "The first step of training is the generation of a training table from your predictive query. You can specify a granular plan to determine how exactly this is done, including specifications of elements like the `split`, `train_start_offset`, and more.\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.pquery.PredictiveQuery.html#kumoai.pquery.PredictiveQuery.generate_training_table).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z6QSlB0dOzUT",
        "outputId": "cc5e31eb-94f6-4429-cb6a-859595461c31"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "gen-traintable-job-415cddd17de24b6092622dd2ef77f34b\n"
          ]
        }
      ],
      "source": [
        "# Launch an asynchronous (nonblocking) job to generate a training table, given\n",
        "# our specified model plan.\n",
        "train_table_job = query.generate_training_table(\n",
        "    query.suggest_training_table_plan(),\n",
        "    non_blocking=True)\n",
        "# The ID of this job:\n",
        "print(train_table_job.id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N59wlP_VqlQq"
      },
      "source": [
        "### Training\n",
        "\n",
        "After launching a training table generation job, we are ready to train a model. Following the same pattern as with training table generation, let's let Kumo intelligently suggest a model plan, that we can modify downstream:\n",
        "\n",
        "<img src=\"https://kumo-sdk-public.s3.us-west-2.amazonaws.com/kumo_training.png\" alt=\"drawing\" width=\"700\"/>\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.trainer.Trainer.html).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yfEjUcA4qpJ2"
      },
      "outputs": [],
      "source": [
        "# Let Kumo intelligently suggest a modeling plan, given the\n",
        "# specified graph and query:\n",
        "model_plan = query.suggest_model_plan(run_mode=RunMode.FAST)\n",
        "\n",
        "# Select `shallow-rhs` architecture\n",
        "model_plan.model_architecture.module = 'embedding'\n",
        "\n",
        "# We may want to limit a model to a single embedding output setting\n",
        "# model_plan.model_architecture.output_embedding_dim = [64]\n",
        "\n",
        "# Increase the number of 1st hop neighbors (user -> rating)\n",
        "model_plan.neighbor_sampling.num_neighbors = [[64, 12]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295,
          "referenced_widgets": [
            "1ae53441c7d14dee9fc33ab3d79d196e",
            "1008363675a247be81bba7d3c01f1be1",
            "464b882557d54460ac21edacaf17f71d",
            "22917cce39084970af4b96ed528d5bd9",
            "573193eee67d45fab4cbec1b8a1bbd29",
            "f42003c576a94c30a27025ba844b172d",
            "f6d419bbe36a49ec9c0f6233f7f4458b",
            "72e85aa117e4451699a13c135a158f7a",
            "d47b0f37e0f34fc88c8234d04f7db0a7",
            "375a3584d9924a1dab59842c4204c7cd",
            "c48e1ea5ff10416186a965c056535ca3"
          ]
        },
        "id": "DYrAJB-XcI-R",
        "outputId": "8dabda5a-d21f-4a78-8e6c-3ca3019169b9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The ID of our training job is trainingjob-8931b11ab4bc45529e61d9af7278a3fe. To see the results later you can run kumo.TrainingJob(\"trainingjob-8931b11ab4bc45529e61d9af7278a3fe\").result()\n",
            "Attaching to training job trainingjob-8931b11ab4bc45529e61d9af7278a3fe. To track this job in the Kumo UI, please visit https://demo-sdk.kumoai.cloud/jobs/training/trainingjob-8931b11ab4bc45529e61d9af7278a3fe. To detach from this job, please enter Ctrl+C: the job will continue to run, and you can re-attach anytime by calling the `attach()` method on the `TrainingJob` object. For example: kumoai.TrainingJob(\"trainingjob-8931b11ab4bc45529e61d9af7278a3fe\").attach()\n",
            "Waiting for job to start.\n",
            "Current stage: Ingesting Data. In progress... Done.\n",
            "Current stage: Waiting for Training Data to be generated. In progress... Done.\n",
            "Current stage: Data Materialization. In progress... Done.\n",
            "Current stage: Provisioning Resources. In progress... Done.\n",
            "Current stage: Loading Materialized Data. In progress... Done.\n",
            "Current stage: Training. In progress..."
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1ae53441c7d14dee9fc33ab3d79d196e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Training:   0%|          |% done "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "<kumoai.trainer.job.TrainingJobResult at 0x7d9ea712c910>"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# A Trainer object manages the execution of a training pipeline, according to\n",
        "# the `model_plan` specification:\n",
        "trainer = kumo.Trainer(model_plan)\n",
        "\n",
        "# Launch an asynchronous (nonblocking) job to train a model, given\n",
        "# our specified model plan. This job is scheduled and orchestrated by the\n",
        "# Kumo platform, and is chained with the job to generate the training table\n",
        "# launched above (it will sequence itself after training table generation is\n",
        "# complete):\n",
        "training_job = trainer.fit(\n",
        "\tgraph=graph,\n",
        "\ttrain_table=train_table_job,\n",
        "\tnon_blocking=True,\n",
        ")\n",
        "\n",
        "# The ID of this job:\n",
        "print(f'The ID of our training job is {training_job.job_id}. To see the results later you can run kumo.TrainingJob(\"{training_job.job_id}\").result()')\n",
        "\n",
        "# Let's follow along...\n",
        "training_job.attach()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sYUmjPhqrzOY"
      },
      "source": [
        "### Aside: Link prediction embedding models\n",
        "Within the graph machine learning framework it's very easy to build a model which can embed two types of entities in the same embedding space by treating the problem as a link prediction task.\n",
        "\n",
        "In essence, Kumo represents the relational data as a heterogeneous graph, and trains the model to predict which nodes will connect to other nodes in the future (or which connections are missing, if the graph is static in time).\n",
        "\n",
        "There are three components to this:\n",
        "- LHS entities/nodes = `user`\n",
        "- RHS entities/nodes = `movie`\n",
        "- Interaction = `user` rates a `movie`\n",
        "\n",
        "This formulation is very flexible, if we had other interactions in the graph we could very well use those, for example watch events, likes, etc. Alternatively, we can identify other pairs of node types to produce embeddigns for, so long as we have a valuable signal of interaction between them.\n",
        "\n",
        "#### Model architecture\n",
        "Kumo provides several state-of-the art Graph Transformer architectures for link prediction [tasks](https://kumo.ai/docs/task-types#common-task-types). Namely, `two-tower`, `shallow-rhs` and `contextGNN` (link to [paper](https://arxiv.org/abs/2411.19513)) architectures. Each comes with their own advantages and disadvantages.\n",
        "\n",
        "We will be using a `shallow-rhs` embedding architecture, which learns ID embeddings for each RHS entity (`movie`). The advantage is that we obtain highly expressive embeddings with little computational expense, but the drawback is that the model is `transductive`.\n",
        "\n",
        "\n",
        "<img src=\"https://kumo-sdk-public.s3.us-west-2.amazonaws.com/shallow_rhs.png\" alt=\"drawing\" width=\"400\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OkxAqHEEIN5S"
      },
      "source": [
        "## Generating Embeddings\n",
        "\n",
        "After our training job is completed, we can generate batch embeddings using our trained model. We can choose to output these batch embeddings directly to a connector (e.g. Amazon S3, Databricks, Snowflake), or we can generate embeddings for download and export at our convenience later with the [`export`](https://kumo-ai.github.io/kumo-sdk/docs/generated/kumoai.trainer.BatchPredictionJobResult.html#kumoai.trainer.BatchPredictionJobResult.export) method.\n",
        "\n",
        "We will do the latter here.\n",
        "\n",
        "*See documentation [here](https://kumo-ai.github.io/kumo-sdk/docs/modules/trainer.html#batch-prediction).*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V3sfzARwOhPM"
      },
      "outputs": [],
      "source": [
        "# specify output connector\n",
        "output_connector = kumo.S3Connector(root_dir='s3://kumo-public-datasets/movielens/embeddings/')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OtSH3A-lJaMf"
      },
      "outputs": [],
      "source": [
        "# past finished training job\n",
        "# training_job_id = 'trainingjob-7b62748768584fa9b5bf35016544d021'\n",
        "# trainer = kumo.Trainer.load(training_job_id)\n",
        "# query = kumo.PredictiveQuery.load_from_training_job(training_job_id)\n",
        "# graph = query.graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0_wnyGHtIZBO",
        "outputId": "ed339ff2-744f-4967-9b05-ca43d6442f9e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Attaching to batch prediction job bp-job-6259fedd1916488385287525c7ee5af8. To track this job in the Kumo UI, please visit https://demo-sdk.kumoai.cloud/jobs/prediction/bp-job-6259fedd1916488385287525c7ee5af8. To detach from this job, please enter Ctrl+C (the job will continue to run, and you can re-attach anytime).\n",
            "Waiting for job to start.\n",
            "Current stage: Ingesting Data. In progress... Done.\n",
            "Current stage: Waiting for Prediction Data to be generated. In progress... Done.\n",
            "Current stage: Data Materialization. In progress... Done.\n",
            "Current stage: Provisioning Resources. In progress... Done.\n",
            "Current stage: Loading Materialized Data. In progress... Done.\n",
            "Current stage: Predicting. In progress..."
          ]
        },
        {
          "data": {
            "text/plain": [
              "<kumoai.trainer.job.BatchPredictionJobResult at 0x7d9ee258ead0>"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "embedding_job = trainer.predict(\n",
        "  graph=graph,\n",
        "  prediction_table=query.generate_prediction_table(non_blocking=True),\n",
        "  training_job_id=training_job.job_id,  # use our training job's model\n",
        "  non_blocking=True,\n",
        "  output_config=OutputConfig(\n",
        "    output_types={'embeddings'},\n",
        "    output_connector=output_connector,\n",
        "    output_table_name='user_movie_test1'\n",
        "  )\n",
        ")\n",
        "embedding_job.attach()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OrsD4gNTOCMt"
      },
      "source": [
        "# Part 2: Using Kumo embeddings with Pinecone to personalize user experience\n",
        "\n",
        "In this second part of the notebook we will focus on the steps that follow after you build your model in Kumo. Okay, so we have a model that can map both `users` and `movies` into the same embedding space. But what can we do with it?\n",
        "\n",
        "Well, if our model does a good job this space represents a very useful tool, because users will be close in this space (as measured by our chosen `similarity_function`) to movies they are likely to interact with. By extension similar `users` will be close to each other and similar `movies` will be close to each other in this embedding space.\n",
        "\n",
        "Above propery opens up a world of possibilities for us, we can use these three notions of similarity to build many personalization products:\n",
        "- Recommend movies to users\n",
        "- Find similar users\n",
        "- Find similar movies\n",
        "All by building a single model! Now only if we had a tool which makes it easy to query this space, i.e. given a user or movie can we find the embeddings closest to it. This is where Pinecone comes in, pinecone makes vector databases simple to understand!\n",
        "\n",
        "## What's below\n",
        "Below we will show some examples of storing and using existing embeddings (like our Kumo embedding!) in Pinecone. We will show examples of:\n",
        "- Finding personalized movies for a user\n",
        "- Finding similar movies\n",
        "- An example of a hybrid search implementation, which allows us to personalize keyword search for each user!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvejSC_HTNdD"
      },
      "source": [
        "## Install, Authenticate, and Initialize Pinecone\n",
        "Now that we can go ahead and initialize the pinecone client."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qvmCIGPYJhOt",
        "outputId": "13d87745-917e-4e03-e257-017580d01b43"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting pinecone\n",
            "  Downloading pinecone-6.0.2-py3-none-any.whl.metadata (9.0 kB)\n",
            "Requirement already satisfied: certifi>=2019.11.17 in /usr/local/lib/python3.11/dist-packages (from pinecone) (2025.1.31)\n",
            "Collecting pinecone-plugin-interface<0.0.8,>=0.0.7 (from pinecone)\n",
            "  Downloading pinecone_plugin_interface-0.0.7-py3-none-any.whl.metadata (1.2 kB)\n",
            "Requirement already satisfied: python-dateutil>=2.5.3 in /usr/local/lib/python3.11/dist-packages (from pinecone) (2.8.2)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.11/dist-packages (from pinecone) (4.12.2)\n",
            "Requirement already satisfied: urllib3>=1.26.0 in /usr/local/lib/python3.11/dist-packages (from pinecone) (2.3.0)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.5.3->pinecone) (1.17.0)\n",
            "Downloading pinecone-6.0.2-py3-none-any.whl (421 kB)\n",
            "\u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m421.9/421.9 kB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pinecone_plugin_interface-0.0.7-py3-none-any.whl (6.2 kB)\n",
            "Installing collected packages: pinecone-plugin-interface, pinecone\n",
            "Successfully installed pinecone-6.0.2 pinecone-plugin-interface-0.0.7\n",
            "Collecting pinecone_notebooks\n",
            "  Downloading pinecone_notebooks-0.1.1-py3-none-any.whl.metadata (2.6 kB)\n",
            "Downloading pinecone_notebooks-0.1.1-py3-none-any.whl (7.3 kB)\n",
            "Installing collected packages: pinecone_notebooks\n",
            "Successfully installed pinecone_notebooks-0.1.1\n"
          ]
        }
      ],
      "source": [
        "!pip install pinecone\n",
        "!pip install pinecone_notebooks\n",
        "# !pip install s3fs\n",
        "# !pip install boto3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 247
        },
        "id": "3u1rQ39-Ji-w",
        "outputId": "9e41e18e-f20d-40e3-fd83-f36b896387ea"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<script type=\"text/javascript\" src=\"https://connect.pinecone.io/embed.js\"></script>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pinecone\n",
        "from pinecone import Pinecone, ServerlessSpec\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from IPython.display import HTML, display\n",
        "\n",
        "import os\n",
        "import itertools\n",
        "\n",
        "if not os.environ.get(\"PINECONE_API_KEY\"):\n",
        "    from pinecone_notebooks.colab import Authenticate\n",
        "    Authenticate()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oN5QkaaXS_xv"
      },
      "outputs": [],
      "source": [
        "# Initialize client\n",
        "pc = Pinecone(api_key=os.environ.get(\"PINECONE_API_KEY\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g7JB8MWVJrKm"
      },
      "source": [
        "## Game plan\n",
        "In order to implement the personalization solutions we mentioned earlier we will perform the following steps:\n",
        "1. Load the `user` and `movie` embeddings\n",
        "2. Join embeddings with the `movie` metadata\n",
        "3. Create Pinecone index for both `users` and `movies`\n",
        "4. Upsert data into indices\n",
        "5. Demonstrate personalized movie recommendation\n",
        "6. Demonstrate finding similar movies"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qJ-GQdmTclK"
      },
      "source": [
        "## Prepare data\n",
        "In the previous part of this notebook we used Kumo to generate the embeddings for both `users` and `movies` in the MovieLens dataset. We have exported the embeddings to S3.\n",
        "\n",
        "We will now load the embeddings. NOTE: the embedding output has the following schema `['ID', 'TYPE', 'EMBEDDING']`, where `TYPE` is either `users` or `movies`. We will first separate the embeddings into separate dataframes. Because we are working with fairly small data we will do all our work locally.\n",
        "\n",
        "\n",
        "First we load the data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vW1wmeZ1Kq79"
      },
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M6Y8QBKnT7BP"
      },
      "outputs": [],
      "source": [
        " # Read the output of the Kumo embedding job\n",
        "embeddings = pd.read_parquet('s3://kumo-demo-datasets/movielens/embeddings/user_movie_embeddings.parquet/')\n",
        "movies = pd.read_parquet('s3://kumo-demo-datasets/movielens/basic_with_posters/movies.parquet')\n",
        "ratings = pd.read_parquet('s3://kumo-demo-datasets/movielens/basic_with_posters/ratings.parquet/')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYannMwfLLQ5"
      },
      "source": [
        "Split the user and movie embeddings into separate dataframes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y81NUORHc0ID",
        "outputId": "332a1617-0745-4a85-d237-4d2414a40cfa"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of user embeddings: 5000\n",
            "Number of movie embeddings: 10269\n"
          ]
        }
      ],
      "source": [
        "def get_user_and_movie_embeddings(df):\n",
        "  \"\"\"\n",
        "  Given a dataframe with columns ['ID', 'TYPE', 'EMBEDDING'],\n",
        "  return two dataframes: one for user embeddings (with ID renamed to 'userId')\n",
        "  and one for movie embeddings (with ID renamed to 'movieId').\n",
        "  \"\"\"\n",
        "  # Separate rows\n",
        "  user_df = df[df['TYPE'] == 'users'].copy()\n",
        "  movie_df = df[df['TYPE'] == 'movies'].copy()\n",
        "\n",
        "  # Rename columns\n",
        "  user_df.rename(columns={'ID': 'userId'}, inplace=True)\n",
        "  movie_df.rename(columns={'ID': 'movieId'}, inplace=True)\n",
        "\n",
        "  # Reset the index\n",
        "  user_df.reset_index(drop=True, inplace=True)\n",
        "  movie_df.reset_index(drop=True, inplace=True)\n",
        "\n",
        "  return user_df[['userId', 'EMBEDDING']], movie_df[['movieId', 'EMBEDDING']]\n",
        "\n",
        "user_embs, movie_embs = get_user_and_movie_embeddings(embeddings)\n",
        "\n",
        "# randomly sample 1k users\n",
        "user_embs = user_embs.sample(5000)\n",
        "\n",
        "print(\"Number of user embeddings:\", len(user_embs))\n",
        "print(\"Number of movie embeddings:\", len(movie_embs))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BnOdt3WuZy4f"
      },
      "outputs": [],
      "source": [
        "# merge movie embeddings with other metadata\n",
        "movie_embs = pd.merge(\n",
        "    movie_embs,\n",
        "    movies[['title', 'genres', 'poster', 'movieId']],\n",
        "    on=\"movieId\",\n",
        "    how=\"left\"\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "3nYUFDI_lASp",
        "outputId": "0e1d80d0-f040-4c2e-9f64-f3d935b1ea67"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"user_embs\",\n  \"rows\": 5000,\n  \"fields\": [\n    {\n      \"column\": \"userId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5000,\n        \"samples\": [\n          \"17695\",\n          \"63733\",\n          \"61809\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"EMBEDDING\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "user_embs"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-c90e7b7e-fff0-4ac5-96fd-d5612fadedbb\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "      <th>EMBEDDING</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>29281</th>\n",
              "      <td>126875</td>\n",
              "      <td>[0.4089927, 0.15596591, -0.35514998, 0.6558638...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26971</th>\n",
              "      <td>31070</td>\n",
              "      <td>[0.08041377, 0.057336926, -0.4774758, 0.145147...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25483</th>\n",
              "      <td>5393</td>\n",
              "      <td>[0.31681156, 0.25866848, -0.5244701, 0.2225395...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19370</th>\n",
              "      <td>6661</td>\n",
              "      <td>[0.3729183, 0.10436365, -0.38383663, 0.5170867...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36097</th>\n",
              "      <td>153855</td>\n",
              "      <td>[0.28993806, -0.017835677, -0.67720664, 0.5017...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c90e7b7e-fff0-4ac5-96fd-d5612fadedbb')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c90e7b7e-fff0-4ac5-96fd-d5612fadedbb button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c90e7b7e-fff0-4ac5-96fd-d5612fadedbb');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-42b7ca5d-370e-467b-bc45-661978192465\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-42b7ca5d-370e-467b-bc45-661978192465')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-42b7ca5d-370e-467b-bc45-661978192465 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       userId                                          EMBEDDING\n",
              "29281  126875  [0.4089927, 0.15596591, -0.35514998, 0.6558638...\n",
              "26971   31070  [0.08041377, 0.057336926, -0.4774758, 0.145147...\n",
              "25483    5393  [0.31681156, 0.25866848, -0.5244701, 0.2225395...\n",
              "19370    6661  [0.3729183, 0.10436365, -0.38383663, 0.5170867...\n",
              "36097  153855  [0.28993806, -0.017835677, -0.67720664, 0.5017..."
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "user_embs.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "jPDTYwinecW2",
        "outputId": "456e5fbe-0feb-4a62-a8d5-45449e5a0ab0"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"movie_embs\",\n  \"rows\": 10269,\n  \"fields\": [\n    {\n      \"column\": \"movieId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10269,\n        \"samples\": [\n          \"197283\",\n          \"182853\",\n          \"144424\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"EMBEDDING\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"title\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10244,\n        \"samples\": [\n          \"Bears (2014)\",\n          \"X - The eXploited (2018)\",\n          \"Tim Maia (2014)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"genres\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 585,\n        \"samples\": [\n          \"Comedy|Fantasy|Horror|Mystery\",\n          \"Action|Comedy|War\",\n          \"Thriller|Western\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"poster\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10229,\n        \"samples\": [\n          \"https://m.media-amazon.com/images/M/MV5BOTI5NTMxODUzMF5BMl5BanBnXkFtZTgwODE4ODIxOTE@._V1_SX300.jpg\",\n          \"https://m.media-amazon.com/images/M/MV5BZGUxNmEzNGEtMDZjNi00OTQ0LWJmMTItMWMyNzg0N2M0NjBkXkEyXkFqcGdeQXVyMjI3NDAyNg@@._V1_SX300.jpg\",\n          \"https://m.media-amazon.com/images/M/MV5BZjZmZWVmOGItNjIwMC00ODNmLWE4MDQtYmY0ZjdlN2EzMTYzXkEyXkFqcGdeQXVyMjA0MzYwMDY@._V1_SX300.jpg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "movie_embs"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-1b6c54f5-5627-43ae-bde7-5a688dcb4366\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>EMBEDDING</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>poster</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>92196</td>\n",
              "      <td>[-0.692878, -0.17644723, 0.18953605, -0.536136...</td>\n",
              "      <td>Crazy Horse (2011)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMjE5NG...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>92374</td>\n",
              "      <td>[-0.094674215, 0.57503706, 0.19659965, -0.8011...</td>\n",
              "      <td>Yolki 2 (2011)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMDg2MW...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>92643</td>\n",
              "      <td>[0.7598486, 1.0139743, 0.8244689, -1.1054865, ...</td>\n",
              "      <td>Monsieur Lazhar (2011)</td>\n",
              "      <td>Children|Comedy|Drama</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BNjM0NT...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>93552</td>\n",
              "      <td>[-0.85924, -0.331058, 0.50398755, -0.081984386...</td>\n",
              "      <td>Blind (Beul-la-in-deu) (2011)</td>\n",
              "      <td>Drama|Horror|Thriller</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMTkzNz...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>93752</td>\n",
              "      <td>[-0.6153211, -0.20514478, 0.42778763, -0.26957...</td>\n",
              "      <td>Saving Face (2012)</td>\n",
              "      <td>Documentary|Drama</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BNThkOG...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1b6c54f5-5627-43ae-bde7-5a688dcb4366')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1b6c54f5-5627-43ae-bde7-5a688dcb4366 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1b6c54f5-5627-43ae-bde7-5a688dcb4366');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e1c6b1fc-3789-4fe8-af07-504210a46048\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e1c6b1fc-3789-4fe8-af07-504210a46048')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e1c6b1fc-3789-4fe8-af07-504210a46048 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "  movieId                                          EMBEDDING  \\\n",
              "0   92196  [-0.692878, -0.17644723, 0.18953605, -0.536136...   \n",
              "1   92374  [-0.094674215, 0.57503706, 0.19659965, -0.8011...   \n",
              "2   92643  [0.7598486, 1.0139743, 0.8244689, -1.1054865, ...   \n",
              "3   93552  [-0.85924, -0.331058, 0.50398755, -0.081984386...   \n",
              "4   93752  [-0.6153211, -0.20514478, 0.42778763, -0.26957...   \n",
              "\n",
              "                           title                 genres  \\\n",
              "0             Crazy Horse (2011)            Documentary   \n",
              "1                 Yolki 2 (2011)                 Comedy   \n",
              "2         Monsieur Lazhar (2011)  Children|Comedy|Drama   \n",
              "3  Blind (Beul-la-in-deu) (2011)  Drama|Horror|Thriller   \n",
              "4             Saving Face (2012)      Documentary|Drama   \n",
              "\n",
              "                                              poster  \n",
              "0  https://m.media-amazon.com/images/M/MV5BMjE5NG...  \n",
              "1  https://m.media-amazon.com/images/M/MV5BMDg2MW...  \n",
              "2  https://m.media-amazon.com/images/M/MV5BNjM0NT...  \n",
              "3  https://m.media-amazon.com/images/M/MV5BMTkzNz...  \n",
              "4  https://m.media-amazon.com/images/M/MV5BNThkOG...  "
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "movie_embs.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cXN6_Bb4edgH"
      },
      "source": [
        "### Store `user` and `movie` embeddings in dense indices\n",
        "\n",
        "A `pinecone` dense index stores dense vectors, which are numerical representations of the meaning and relationships of text, images, or other types of data. If you use an external embedding model (like Kumo) to convert your data to dense vectors, create a dense index as follows:\n",
        "\n",
        "Since we've already produced embeddings of both `users` and `movies` we won't be using any [integrated embedding models](https://docs.pinecone.io/reference/api/2025-01/control-plane/create_for_model) that Pinecone provides. Instead we need to create a [dense index](https://docs.pinecone.io/reference/api/2025-01/control-plane/create_index) by providing doing the following:\n",
        "- Provide a `name` for the index.\n",
        "- Set the `vector_type` to `dense`.\n",
        "- Specify the `dimension` and similarity `metric` of the vectors you\u2019ll store in the index. This should match the dimension and metric supported by your embedding model.\n",
        "- Set `spec.cloud` and `spec.region` to the cloud and region where the index should be deployed. Import the `ServerlessSpec` class, since we're working with Python.\n",
        "\n",
        "Other parameters are optional. See the [API reference](https://docs.pinecone.io/reference/api/2025-01/control-plane/create_index) for details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gTo9b0_Cgp42"
      },
      "outputs": [],
      "source": [
        "# index names\n",
        "movie_index_name = \"movies-index\"\n",
        "user_index_name = \"users-index\"\n",
        "\n",
        "embedding_dim = len(movie_embs['EMBEDDING'].iloc[0])\n",
        "\n",
        "# Create movie index if it doesn't exist\n",
        "if not pc.has_index(movie_index_name):\n",
        "  pc.create_index(\n",
        "    name=movie_index_name,\n",
        "    dimension=embedding_dim,\n",
        "    metric='dotproduct',\n",
        "    spec=ServerlessSpec(\n",
        "      cloud=\"aws\",\n",
        "      region=\"us-east-1\"\n",
        "    ),\n",
        "  )\n",
        "\n",
        "# Create user index if it doesn't exist\n",
        "if not pc.has_index(user_index_name):\n",
        "  pc.create_index(\n",
        "    name=user_index_name,\n",
        "    dimension=embedding_dim,\n",
        "    metric='dotproduct',\n",
        "    spec=ServerlessSpec(\n",
        "      cloud=\"aws\",\n",
        "      region=\"us-east-1\"\n",
        "    ),\n",
        "  )\n",
        "\n",
        "# Now connect to these indices\n",
        "movie_index = pc.Index(movie_index_name)\n",
        "user_index = pc.Index(user_index_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSvQPjT2gzZX"
      },
      "source": [
        "### Upserting our data into the `user` index and `movie` index\n",
        "We have created both the `user` and the `movie` index and we can proceed with upserting the embeddings into each index. An `Upsert` is the basic operation of adding (dense) data to Pinecone indices. Alternatively, we could also use `import` if we were dealing with very large data.\n",
        "\n",
        "In order to upsert our own data into a dense index, we need to:\n",
        "\n",
        "- Specify the `namespace` to upsert into. If the namespace doesn\u2019t exist, it is created. To use the default namespace, set the namespace to an empty string (`\"\"`).\n",
        "- Format your input data as records, each with the following:\n",
        "  - An `id` field with a unique record identifier for the index namespace.\n",
        "  - A `values` field with the dense vector values.\n",
        "  - Optionally, a `metadata` field with key-value pairs to store additional information or context. This alows for Metadata Filtering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Up4fwwY5lBoS"
      },
      "outputs": [],
      "source": [
        "# transform movie_embs to pinecone friendly format\n",
        "movie_embs_pc = pd.DataFrame({\n",
        "    'id': movie_embs['movieId'],\n",
        "    'values': movie_embs['EMBEDDING'],\n",
        "    'metadata': movie_embs[['title', 'genres', 'poster', 'movieId']].to_dict(orient='records')\n",
        "})\n",
        "\n",
        "# transform user_embs to pinecone friendly format\n",
        "user_embs_pc = pd.DataFrame({\n",
        "    'id': user_embs['userId'],\n",
        "    'values': user_embs['EMBEDDING']\n",
        "})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67,
          "referenced_widgets": [
            "c264887ae1fc44248582ab0be1820a1e",
            "8b66f6ce8b85445e95bbb80a94e4deb8",
            "4865ca9629354009b10dfc3dbfbe8cb0",
            "36305d3aaaa749448960a347e1235cf9",
            "0dfbcc127588461baecbae6bbda89ffa",
            "10861440e9b148d6945de19760c096a5",
            "2201a6a170764afb9a9dae7e0a332d4c",
            "1f2d987665f649ce8f31154404adbe7a",
            "705a070e70cd4de9b7c3150424eaecd8",
            "7754cea27d234822aba04abc06174058",
            "da59d89258fa4028909077eb36c00ab0"
          ]
        },
        "id": "ovyI5li7jGX6",
        "outputId": "46984664-5de9-4cbe-93aa-c0719f762f6a"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c264887ae1fc44248582ab0be1820a1e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "sending upsert requests:   0%|          | 0/10269 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'upserted_count': 10269}"
            ]
          },
          "execution_count": 62,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# upsert movies\n",
        "movie_index.upsert_from_dataframe(\n",
        "    df=movie_embs_pc,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 67,
          "referenced_widgets": [
            "0b41ae923b314bfea177ca45463f62b6",
            "0fb5df25c1284449b31b895e25ffccdb",
            "2f6d651f64994baf8ba560605144582b",
            "09472642628a476ead5161d459b851fc",
            "de5d2be5f8ce47aca6da871f46349220",
            "04937f4e040b436d9f888c48e0cfc058",
            "60949a9127ff4f339bb1a2b3c064bd5e",
            "6bdae7358b924024891dd70fcdc6c565",
            "2f2f0893c7e74370a32cbf156143243b",
            "53e96e46a16c41f6aaefa91e82aad9fb",
            "82815b6d6968437880ed342435924db5"
          ]
        },
        "id": "muZJ6RzwZf4V",
        "outputId": "d2603509-7a8d-486f-d007-ffca84e02b15"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "0b41ae923b314bfea177ca45463f62b6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "sending upsert requests:   0%|          | 0/5000 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'upserted_count': 5000}"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# upsert users\n",
        "user_index.upsert_from_dataframe(\n",
        "    df=user_embs_pc,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IWPPKS_RsN6a"
      },
      "source": [
        "We can perform a quick sanity check to see if the data has been upserted by fetching a movie entry:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oJTaFKTUkZDI",
        "outputId": "e68d6e09-5f18-402f-bac7-3e8228a6b570"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "FetchResponse(namespace='', vectors={'92196': Vector(id='92196', values=[-0.692878, -0.176447228, 0.18953605, -0.536136508, -0.249925777, -0.560492575, -0.0347835496, 0.0663761422, 0.246170118, -0.397063106, 0.137878582, 0.452405483, 0.329483032, -0.295125395, -0.267708063, 0.11085692, 0.235440329, 0.0689748898, 0.330064863, -0.245473638, 0.255116254, -0.103964798, 0.0833421797, 0.353449434, -0.38890925, 0.299163342, -0.0375199243, 0.360139638, 0.28604418, -0.470369279, 0.586901, 0.172865361, -0.174468979, -0.0898205265, -0.172712594, -0.395742, 0.359849483, 0.124656901, 0.145688295, 0.0534218736, 0.00882671215, 0.131214038, 0.0584079698, 0.145428941, -0.468755722, -0.204533279, -0.0454566777, -0.335710615, 0.947899878, 0.267457813, 0.560013413, -0.461288, 0.458088249, 0.0351200923, 0.159144238, 0.0651222914, 0.0709731132, 0.0506714284, 0.232988834, -0.162132233, 0.327170819, -0.00265975366, 0.116343915, 0.16967009], metadata={'genres': 'Documentary', 'movieId': '92196', 'poster': 'https://m.media-amazon.com/images/M/MV5BMjE5NGQwYjItYjgyMi00ZWJmLThiYzAtOGFiN2NhY2FjMmNhXkEyXkFqcGdeQXVyNDkzNTM2ODg@._V1_SX300.jpg', 'title': 'Crazy Horse (2011)'}, sparse_values=None)}, usage={'read_units': 1})"
            ]
          },
          "execution_count": 65,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "movie_index.fetch(ids=[\"92196\"], namespace=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P7p1LBrksMSz"
      },
      "source": [
        "## Exploring personalized user recommendations\n",
        "To find recommended movies for a particular user we need to:\n",
        "1. Get the `user` embedding via their `userId` from the `users` index\n",
        "2. Use the `user` embedding from 1. to query the `movies` index\n",
        "This will yield movies that are the closest to the user in embedding space, and thus most likely for the user to interact with next.\n",
        "\n",
        "\n",
        "The helper function below fetches a user\u2019s most recently rated movies and then queries Pinecone to find top recommended titles for that user. It displays the user\u2019s rating history with posters and ratings, followed by a row of recommended movies showing their posters and similarity scores. It provides a quick way to visualize both the user\u2019s past interactions and personalized recommendations side by side.\n",
        "\n",
        "To learn more look at:\n",
        "- [`index.fetch`](https://docs.pinecone.io/reference/api/2025-01/data-plane/fetch)\n",
        "- [`index.query`](https://docs.pinecone.io/guides/data/query-data#semantic-search)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hJ4Jx0aImM8R"
      },
      "source": [
        "### HTML Helper function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IRMce23Hqcu1"
      },
      "outputs": [],
      "source": [
        "def display_user_history_and_recommendations(\n",
        "    user_id,\n",
        "    ratings_df,\n",
        "    movie_df,\n",
        "    user_index,\n",
        "    movie_index,\n",
        "    top_k=5,\n",
        "    max_history=12\n",
        "):\n",
        "    \"\"\"\n",
        "    1) Display the user's most recent ratings (up to `max_history`).\n",
        "    2) Display top-K recommended movies for that user in a row below.\n",
        "\n",
        "    Assumes Pinecone's movie index has metadata:\n",
        "      {\n",
        "        'title': <str>,\n",
        "        'poster': <str>,\n",
        "        'genres': <str>\n",
        "      }\n",
        "    \"\"\"\n",
        "    # Filter the user's ratings\n",
        "    user_ratings = ratings_df[ratings_df['userId'] == user_id].copy()\n",
        "    # Sort by timestamp descending (most recent first)\n",
        "    user_ratings.sort_values('timestamp', ascending=False, inplace=True)\n",
        "    # Take up to `max_history` rows\n",
        "    user_ratings = user_ratings.head(max_history)\n",
        "\n",
        "    # Merge with movie metadata so we can show titles/posters\n",
        "    user_ratings = user_ratings.merge(\n",
        "        movie_df[['movieId', 'title', 'poster', 'genres']],\n",
        "        on='movieId',\n",
        "        how='left'\n",
        "    )\n",
        "\n",
        "    # Build HTML for displaying recent ratings\n",
        "    html_str = \"<h4>Recent Ratings</h4><div style='display:flex;flex-wrap:wrap;'>\"\n",
        "    for _, row in user_ratings.iterrows():\n",
        "        title = row.get('title', 'Unknown Title')\n",
        "        poster_url = row.get('poster', '')\n",
        "        rating_val = row.get('rating', 'N/A')\n",
        "\n",
        "        html_str += f\"\"\"\n",
        "        <div style='margin:10px;text-align:center;'>\n",
        "            <img src=\"{poster_url}\" alt=\"{title}\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
        "            <div>{title}</div>\n",
        "            <div>Rating: {rating_val}</div>\n",
        "        </div>\n",
        "        \"\"\"\n",
        "    html_str += \"</div>\"\n",
        "\n",
        "    # Fetch user vector from Pinecone (index holding user embeddings)\n",
        "    fetch_result = user_index.fetch(ids=[user_id])\n",
        "    if fetch_result and fetch_result.vectors:\n",
        "        user_vector = fetch_result.vectors[user_id].values\n",
        "\n",
        "        # Query the movie index\n",
        "        query_response = movie_index.query(\n",
        "            vector=user_vector,\n",
        "            top_k=top_k,\n",
        "            include_metadata=True\n",
        "        )\n",
        "\n",
        "        # Build HTML for displaying recommended movies\n",
        "        html_str += \"<h4>Recommended Movies</h4><div style='display:flex;flex-wrap:wrap;'>\"\n",
        "        for match in query_response.matches:\n",
        "            md = match.metadata or {}\n",
        "            rec_title = md.get('title', 'Unknown Title')\n",
        "            rec_poster = md.get('poster', '')\n",
        "            rec_genres = md.get('genres', '')\n",
        "            rec_score = match.score\n",
        "\n",
        "            html_str += f\"\"\"\n",
        "            <div style='margin:10px;text-align:center;'>\n",
        "                <img src=\"{rec_poster}\" alt=\"{rec_title}\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
        "                <div>{rec_title}</div>\n",
        "                <div>Score: {rec_score:.4f}</div>\n",
        "            </div>\n",
        "            \"\"\"\n",
        "        html_str += \"</div>\"\n",
        "    else:\n",
        "        # If the user isn't found in the user index, no recommendations\n",
        "        html_str += \"<p>No user vector found, cannot generate recommendations.</p>\"\n",
        "\n",
        "    # Display everything\n",
        "    display(HTML(html_str))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y1c1Gzb6meSW"
      },
      "source": [
        "### Playground"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "NIOu5_8SqzU-",
        "outputId": "0523275b-d32b-4851-ff1e-32893688a88b"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"user_embs[['userId']]\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"userId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"120312\",\n          \"145409\",\n          \"154886\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-f997a54e-b34d-46e0-87ba-784ea4e0a39b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>36649</th>\n",
              "      <td>145350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32982</th>\n",
              "      <td>120312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42302</th>\n",
              "      <td>154886</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12977</th>\n",
              "      <td>94440</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40782</th>\n",
              "      <td>145409</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f997a54e-b34d-46e0-87ba-784ea4e0a39b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f997a54e-b34d-46e0-87ba-784ea4e0a39b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f997a54e-b34d-46e0-87ba-784ea4e0a39b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-cb2c8815-89e2-4c87-9d2c-0266d593d4bc\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cb2c8815-89e2-4c87-9d2c-0266d593d4bc')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-cb2c8815-89e2-4c87-9d2c-0266d593d4bc button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       userId\n",
              "36649  145350\n",
              "32982  120312\n",
              "42302  154886\n",
              "12977   94440\n",
              "40782  145409"
            ]
          },
          "execution_count": 67,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "user_embs[['userId']].sample(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10b2pgySNORX"
      },
      "source": [
        "We can use our function to explore recommendations for a particular user, given their movie rating history! Feel free to explore different `query_user_id`'s."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 821
        },
        "id": "QtEf7zLEpy6V",
        "outputId": "1173bc36-3013-4f40-f5b6-58c92c6948c4"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<h4>Recent Ratings</h4><div style='display:flex;flex-wrap:wrap;'>\n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BOTgwMzFiMWYtZDhlNS00ODNkLWJiODAtZDVhNzgyNzJhYjQ4L2ltYWdlXkEyXkFqcGdeQXVyNzEzOTYxNTQ@._V1_SX300.jpg\" alt=\"The Imitation Game (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>The Imitation Game (2014)</div>\n",
              "            <div>Rating: 3.5</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMzM5NjUxOTEyMl5BMl5BanBnXkFtZTgwNjEyMDM0MDE@._V1_SX300.jpg\" alt=\"Grand Budapest Hotel, The (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Grand Budapest Hotel, The (2014)</div>\n",
              "            <div>Rating: 5.0</div>\n",
              "        </div>\n",
              "        </div><h4>Recommended Movies</h4><div style='display:flex;flex-wrap:wrap;'>\n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMTc2MTQ3MDA1Nl5BMl5BanBnXkFtZTgwODA3OTI4NjE@._V1_SX300.jpg\" alt=\"The Martian (2015)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>The Martian (2015)</div>\n",
              "                <div>Score: 8.8813</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMjI0ODcxNzM1N15BMl5BanBnXkFtZTgwMzIwMTEwNDI@._V1_SX300.jpg\" alt=\"Three Billboards Outside Ebbing, Missouri (2017)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Three Billboards Outside Ebbing, Missouri (2017)</div>\n",
              "                <div>Score: 8.0932</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMjMxNjY2MDU1OV5BMl5BanBnXkFtZTgwNzY1MTUwNTM@._V1_SX300.jpg\" alt=\"Avengers: Infinity War - Part I (2018)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Avengers: Infinity War - Part I (2018)</div>\n",
              "                <div>Score: 8.0619</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BOTgxMDQwMDk0OF5BMl5BanBnXkFtZTgwNjU5OTg2NDE@._V1_SX300.jpg\" alt=\"Inside Out (2015)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Inside Out (2015)</div>\n",
              "                <div>Score: 8.0450</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BOTA5NDZlZGUtMjAxOS00YTRkLTkwYmMtYWQ0NWEwZDZiNjEzXkEyXkFqcGdeQXVyMTMxODk2OTU@._V1_SX300.jpg\" alt=\"Whiplash (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Whiplash (2014)</div>\n",
              "                <div>Score: 8.0119</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMTExMzU0ODcxNDheQTJeQWpwZ15BbWU4MDE1OTI4MzAy._V1_SX300.jpg\" alt=\"Arrival (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Arrival (2016)</div>\n",
              "                <div>Score: 7.7874</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMTAwMjU5OTgxNjZeQTJeQWpwZ15BbWU4MDUxNDYxODEx._V1_SX300.jpg\" alt=\"Guardians of the Galaxy (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Guardians of the Galaxy (2014)</div>\n",
              "                <div>Score: 7.6403</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BYjQ5NjM0Y2YtNjZkNC00ZDhkLWJjMWItN2QyNzFkMDE3ZjAxXkEyXkFqcGdeQXVyODIxMzk5NjA@._V1_SX300.jpg\" alt=\"Coco (2017)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Coco (2017)</div>\n",
              "                <div>Score: 7.6226</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMjA1MTc1NTg5NV5BMl5BanBnXkFtZTgwOTM2MDEzNzE@._V1_SX300.jpg\" alt=\"The Hateful Eight (2015)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>The Hateful Eight (2015)</div>\n",
              "                <div>Score: 7.5636</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BN2U1YzdhYWMtZWUzMi00OWI1LWFkM2ItNWVjM2YxMGQ2MmNhXkEyXkFqcGdeQXVyNjU0OTQ0OTY@._V1_SX300.jpg\" alt=\"Nightcrawler (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Nightcrawler (2014)</div>\n",
              "                <div>Score: 7.4994</div>\n",
              "            </div>\n",
              "            </div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# For demonstration, pick a user ID that exists in your user index and ratings table\n",
        "query_user_id = \"108263\"\n",
        "\n",
        "display_user_history_and_recommendations(\n",
        "    user_id=query_user_id,\n",
        "    ratings_df=ratings,\n",
        "    movie_df=movies,\n",
        "    user_index=user_index,\n",
        "    movie_index=movie_index,\n",
        "    top_k=10,\n",
        "    max_history=10\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g9H0hmqmp6cw"
      },
      "source": [
        "## Finding similar movies\n",
        "We can repeat a similar excercise for finding similar movies. Except we now:\n",
        "1. Get the `movie` embedding via their `movieId` from the `movie` index\n",
        "2. Use the `movie` embedding from 1. to query the `movies` index\n",
        "This will yield movies that are the closest to the query movie in embedding space, and are most likely to be interacted with by simliar users.\n",
        "\n",
        "The helper function takes a given \u201canchor\u201d movie, retrieves its embedding from Pinecone, and queries for similar movie embeddings. It displays the anchor movie\u2019s details (title, poster, genres) followed by the top-K similar movies with their posters, genres, and similarity scores. It\u2019s a handy way to visualize content-based recommendations or \u201cmore like this\u201d experiences within a notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NvaakFjumkQ-"
      },
      "source": [
        "#### HTML Helper"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UdwdZd5esdeV"
      },
      "outputs": [],
      "source": [
        "def display_movie_and_similar(\n",
        "    movie_id,\n",
        "    movie_df,\n",
        "    movie_index,\n",
        "    top_k=5\n",
        "):\n",
        "    \"\"\"\n",
        "    1) Display the anchor movie's details (poster, title, genres).\n",
        "    2) Display top-K similar movies by querying the same movie_index.\n",
        "\n",
        "    Assumes:\n",
        "    - movie_df has columns [movieId, title, poster, genres] (adjust as needed).\n",
        "    - movie_index is a Pinecone index with embeddings for each movie.\n",
        "    - The Pinecone metadata is stored as:\n",
        "        {\n",
        "          'title': <str>,\n",
        "          'poster': <str>,\n",
        "          'genres': <str>\n",
        "        }\n",
        "    \"\"\"\n",
        "    # Retrieve anchor movie info from local DataFrame\n",
        "    # (Alternatively, you could fetch from Pinecone if you store full metadata there.)\n",
        "    anchor_movie = movie_df[movie_df['movieId'] == movie_id].head(1)\n",
        "\n",
        "    # Build HTML for the anchor movie\n",
        "    html_str = \"<h4>Anchor Movie</h4>\"\n",
        "    if len(anchor_movie) == 0:\n",
        "        # If not found in local DF, just display a warning\n",
        "        html_str += f\"<p>Movie {movie_id} not found in local metadata.</p>\"\n",
        "    else:\n",
        "        row = anchor_movie.iloc[0]\n",
        "        title = row.get('title', 'Unknown Title')\n",
        "        poster_url = row.get('poster', '')\n",
        "        genres = row.get('genres', '')\n",
        "        html_str += f\"\"\"\n",
        "        <div style='margin-bottom:15px;'>\n",
        "            <img src=\"{poster_url}\" alt=\"{title}\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
        "            <div><strong>{title}</strong></div>\n",
        "            <div>Genres: {genres}</div>\n",
        "            <div>Movie ID: {movie_id}</div>\n",
        "        </div>\n",
        "        \"\"\"\n",
        "\n",
        "    # Fetch this movie's vector from Pinecone\n",
        "    fetch_result = movie_index.fetch(ids=[movie_id])\n",
        "    if not fetch_result or not fetch_result.vectors:\n",
        "        html_str += f\"<p>No vector found in Pinecone for movieId '{movie_id}'.</p>\"\n",
        "        display(HTML(html_str))\n",
        "        return\n",
        "\n",
        "    movie_vector = fetch_result.vectors[movie_id].values\n",
        "\n",
        "    # Query for similar movies (include_metadata=True so we can read title, poster, etc.)\n",
        "    query_response = movie_index.query(\n",
        "        vector=movie_vector,\n",
        "        top_k=top_k,\n",
        "        include_metadata=True\n",
        "    )\n",
        "\n",
        "    # Build HTML for similar movies\n",
        "    html_str += \"<h4>Similar Movies</h4><div style='display:flex;flex-wrap:wrap;'>\"\n",
        "    for match in query_response.matches:\n",
        "        md = match.metadata or {}\n",
        "        similar_title = md.get('title', 'Unknown Title')\n",
        "        similar_poster = md.get('poster', '')\n",
        "        similar_genres = md.get('genres', '')\n",
        "        score = match.score\n",
        "\n",
        "        html_str += f\"\"\"\n",
        "        <div style='margin:10px;text-align:center;'>\n",
        "            <img src=\"{similar_poster}\" alt=\"{similar_title}\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
        "            <div>{similar_title}</div>\n",
        "            <div>Score: {score:.4f}</div>\n",
        "        </div>\n",
        "        \"\"\"\n",
        "    html_str += \"</div>\"\n",
        "\n",
        "    # Display everything\n",
        "    display(HTML(html_str))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apbGcr_Mmmng"
      },
      "source": [
        "### Playground"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "id": "o8_rBKJks3_-",
        "outputId": "9ae43153-0e6b-4257-e81b-35049f4cbba0"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"movie_embs\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"movieId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"184453\",\n          \"162768\",\n          \"194947\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"EMBEDDING\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"title\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"He's Way More Famous Than You (2013)\",\n          \"Out of Love (2016)\",\n          \"Cam (2018)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"genres\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"Drama\",\n          \"Sci-Fi\",\n          \"Documentary\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"poster\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"https://m.media-amazon.com/images/M/MV5BMTkyMzc2Njc3M15BMl5BanBnXkFtZTcwOTcyODQzOQ@@._V1_SX300.jpg\",\n          \"https://m.media-amazon.com/images/M/MV5BNTBiNzk5OGItYTJmYy00NjVhLTlmNGQtNDQ3NmZhZDQwNDVlL2ltYWdlL2ltYWdlXkEyXkFqcGdeQXVyNjk4MDMwMDQ@._V1_SX300.jpg\",\n          \"https://m.media-amazon.com/images/M/MV5BYWYwZDg4Y2YtN2RiYS00YTJlLTkyNTctNjJlZjVjMmQzMmNiXkEyXkFqcGdeQXVyMjY5ODI4NDk@._V1_SX300.jpg\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-195d3950-2bd6-46b8-8282-01e4568c2b8e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>EMBEDDING</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>poster</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>5112</th>\n",
              "      <td>166245</td>\n",
              "      <td>[0.7574303, -0.12714083, 0.039902356, 0.949986...</td>\n",
              "      <td>Good Kids (2016)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMTc1Nj...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4615</th>\n",
              "      <td>162768</td>\n",
              "      <td>[-0.077845, -0.34369758, 0.2664674, 0.29583338...</td>\n",
              "      <td>Out of Love (2016)</td>\n",
              "      <td>Drama</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BNTBiNz...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4561</th>\n",
              "      <td>162540</td>\n",
              "      <td>[0.030866267, 0.3192884, -0.015940899, 0.45453...</td>\n",
              "      <td>Amateur Night (2016)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BZTQ2N2...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4682</th>\n",
              "      <td>163468</td>\n",
              "      <td>[-0.44087893, -0.08144211, 0.58524895, -0.2222...</td>\n",
              "      <td>Maz Jobrani: I'm Not a Terrorist But I've Play...</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BZTRhMG...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5052</th>\n",
              "      <td>165841</td>\n",
              "      <td>[-0.17871223, -0.14920716, 0.17397414, 0.37675...</td>\n",
              "      <td>All These Sleepless Nights (2016)</td>\n",
              "      <td>Documentary</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMjA0ND...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8952</th>\n",
              "      <td>194947</td>\n",
              "      <td>[-0.023676129, 0.1615332, 0.39900583, -0.20030...</td>\n",
              "      <td>Cam (2018)</td>\n",
              "      <td>Horror|Thriller</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BYWYwZD...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7863</th>\n",
              "      <td>186903</td>\n",
              "      <td>[-0.40621254, -0.5325371, 0.08453919, 0.165804...</td>\n",
              "      <td>Greg Davies: You Magnificent Beast (2018)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMWI1Yz...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>689</th>\n",
              "      <td>112735</td>\n",
              "      <td>[-0.14697719, -0.09704951, -0.10601158, -0.080...</td>\n",
              "      <td>Charlie's Country (2013)</td>\n",
              "      <td>Drama</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMTQ4OT...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7538</th>\n",
              "      <td>184453</td>\n",
              "      <td>[-0.1685566, 0.024316644, 0.31974322, -0.11731...</td>\n",
              "      <td>He's Way More Famous Than You (2013)</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BMTkyMz...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7976</th>\n",
              "      <td>187655</td>\n",
              "      <td>[-0.060699537, -0.14766778, 0.022586348, -0.09...</td>\n",
              "      <td>Genesis: The Fall of Eden (2018)</td>\n",
              "      <td>Sci-Fi</td>\n",
              "      <td>https://m.media-amazon.com/images/M/MV5BZTczNG...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-195d3950-2bd6-46b8-8282-01e4568c2b8e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-195d3950-2bd6-46b8-8282-01e4568c2b8e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-195d3950-2bd6-46b8-8282-01e4568c2b8e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-22cef02f-85dd-4c0f-8c6e-8e7c9c8840ad\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-22cef02f-85dd-4c0f-8c6e-8e7c9c8840ad')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-22cef02f-85dd-4c0f-8c6e-8e7c9c8840ad button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "     movieId                                          EMBEDDING  \\\n",
              "5112  166245  [0.7574303, -0.12714083, 0.039902356, 0.949986...   \n",
              "4615  162768  [-0.077845, -0.34369758, 0.2664674, 0.29583338...   \n",
              "4561  162540  [0.030866267, 0.3192884, -0.015940899, 0.45453...   \n",
              "4682  163468  [-0.44087893, -0.08144211, 0.58524895, -0.2222...   \n",
              "5052  165841  [-0.17871223, -0.14920716, 0.17397414, 0.37675...   \n",
              "8952  194947  [-0.023676129, 0.1615332, 0.39900583, -0.20030...   \n",
              "7863  186903  [-0.40621254, -0.5325371, 0.08453919, 0.165804...   \n",
              "689   112735  [-0.14697719, -0.09704951, -0.10601158, -0.080...   \n",
              "7538  184453  [-0.1685566, 0.024316644, 0.31974322, -0.11731...   \n",
              "7976  187655  [-0.060699537, -0.14766778, 0.022586348, -0.09...   \n",
              "\n",
              "                                                  title           genres  \\\n",
              "5112                                   Good Kids (2016)           Comedy   \n",
              "4615                                 Out of Love (2016)            Drama   \n",
              "4561                               Amateur Night (2016)           Comedy   \n",
              "4682  Maz Jobrani: I'm Not a Terrorist But I've Play...           Comedy   \n",
              "5052                  All These Sleepless Nights (2016)      Documentary   \n",
              "8952                                         Cam (2018)  Horror|Thriller   \n",
              "7863          Greg Davies: You Magnificent Beast (2018)           Comedy   \n",
              "689                            Charlie's Country (2013)            Drama   \n",
              "7538               He's Way More Famous Than You (2013)           Comedy   \n",
              "7976                   Genesis: The Fall of Eden (2018)           Sci-Fi   \n",
              "\n",
              "                                                 poster  \n",
              "5112  https://m.media-amazon.com/images/M/MV5BMTc1Nj...  \n",
              "4615  https://m.media-amazon.com/images/M/MV5BNTBiNz...  \n",
              "4561  https://m.media-amazon.com/images/M/MV5BZTQ2N2...  \n",
              "4682  https://m.media-amazon.com/images/M/MV5BZTRhMG...  \n",
              "5052  https://m.media-amazon.com/images/M/MV5BMjA0ND...  \n",
              "8952  https://m.media-amazon.com/images/M/MV5BYWYwZD...  \n",
              "7863  https://m.media-amazon.com/images/M/MV5BMWI1Yz...  \n",
              "689   https://m.media-amazon.com/images/M/MV5BMTQ4OT...  \n",
              "7538  https://m.media-amazon.com/images/M/MV5BMTkyMz...  \n",
              "7976  https://m.media-amazon.com/images/M/MV5BZTczNG...  "
            ]
          },
          "execution_count": 70,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "movie_embs.sample(10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 805
        },
        "id": "kShvqA4xs239",
        "outputId": "44b7de63-3b64-49ab-8f2b-85f65e7c983d"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<h4>Anchor Movie</h4>\n",
              "        <div style='margin-bottom:15px;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMjAxOTY1MDg2N15BMl5BanBnXkFtZTgwNDQ3OTI0NDE@._V1_SX300.jpg\" alt=\"3 Hearts (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div><strong>3 Hearts (2014)</strong></div>\n",
              "            <div>Genres: Drama</div>\n",
              "            <div>Movie ID: 140096</div>\n",
              "        </div>\n",
              "        <h4>Similar Movies</h4><div style='display:flex;flex-wrap:wrap;'>\n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMjAxOTY1MDg2N15BMl5BanBnXkFtZTgwNDQ3OTI0NDE@._V1_SX300.jpg\" alt=\"3 Hearts (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>3 Hearts (2014)</div>\n",
              "            <div>Score: 8.4746</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMTU3MDUwMzAyNl5BMl5BanBnXkFtZTgwNzIzODMxNzE@._V1_SX300.jpg\" alt=\"Flowers (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Flowers (2014)</div>\n",
              "            <div>Score: 5.9339</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BODM3ODMwMTIxNV5BMl5BanBnXkFtZTgwODc4NTM0MzE@._V1_SX300.jpg\" alt=\"Mr. Kaplan (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Mr. Kaplan (2014)</div>\n",
              "            <div>Score: 5.4586</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMjE1NjU0ODA0NF5BMl5BanBnXkFtZTgwMDQxNzc3NTE@._V1_SX300.jpg\" alt=\"The Thin Yellow Line (2015)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>The Thin Yellow Line (2015)</div>\n",
              "            <div>Score: 5.4128</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BOGMzOTQ5ZTItNmQyMC00YTA1LTgwZDItZjRkN2M5MjllN2FmXkEyXkFqcGdeQXVyNTM0MDc1ODE@._V1_SX300.jpg\" alt=\"We Will Not Get Used To (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>We Will Not Get Used To (2016)</div>\n",
              "            <div>Score: 5.3687</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMTQ3ODY1NDAxNl5BMl5BanBnXkFtZTgwNzcxMDg0MzE@._V1_SX300.jpg\" alt=\"Friends from France (2013)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Friends from France (2013)</div>\n",
              "            <div>Score: 5.3522</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMzcwNTczNDQ5N15BMl5BanBnXkFtZTgwNDIxMDY0NzE@._V1_SX300.jpg\" alt=\"J\u00e4ttil\u00e4inen (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>J\u00e4ttil\u00e4inen (2016)</div>\n",
              "            <div>Score: 5.3162</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMTQ5NzcwNTM3OV5BMl5BanBnXkFtZTgwMDM3NTA1NzE@._V1_SX300.jpg\" alt=\"Misconduct (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Misconduct (2016)</div>\n",
              "            <div>Score: 5.3153</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMjUzOTQwNTQxNF5BMl5BanBnXkFtZTgwNDAwNTk1MjE@._V1_SX300.jpg\" alt=\"Nas: Time Is Illmatic (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Nas: Time Is Illmatic (2014)</div>\n",
              "            <div>Score: 5.2616</div>\n",
              "        </div>\n",
              "        \n",
              "        <div style='margin:10px;text-align:center;'>\n",
              "            <img src=\"https://m.media-amazon.com/images/M/MV5BMTg3Njc1NzM3NV5BMl5BanBnXkFtZTgwMTE4ODI2MzE@._V1_SX300.jpg\" alt=\"Stray Dog (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "            <div>Stray Dog (2014)</div>\n",
              "            <div>Score: 5.2475</div>\n",
              "        </div>\n",
              "        </div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Suppose we have a movieId\n",
        "anchor_movie_id = \"140096\"\n",
        "\n",
        "display_movie_and_similar(\n",
        "    movie_id=anchor_movie_id,\n",
        "    movie_df=movie_embs,\n",
        "    movie_index=movie_index,\n",
        "    top_k=10\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ov8HWctEs7FK"
      },
      "source": [
        "## Hybrid Search: Personalizing user's keyword search\n",
        "But we can go beyond just querying with dense vectors or performing keyword search. Pinecone allows for simple implementations of **hybrid search** which combines both to get the best of both worlds. Here we will show just a single implementation, but if you want to learn more please take a look at the Pinecone documentation on the topic: [Getting Started with Hybrid Search](https://www.pinecone.io/learn/hybrid-search-intro/).\n",
        "\n",
        " The premise is quite straightforward, let's say we are building a streaming service and we would like to provide our users with a search bar to find the next movie to watch. Most user queries will be very simple (e.g. \"Action adventure\") and a simple keyword search can be improved upon as all action adventure movies stil need to be ranked. It turns out we can do this in a personalized way, by combining keyword search with dense similarity between `users` and `movies`.\n",
        "\n",
        " Below is a demonstration of combining:\n",
        " - **Kumo embeddings**: which bring a notion of user<>movie similarity, and\n",
        " - **Keyword search**: which guides the recommendations towards a particular user query"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LgQsrJDLP5Tl"
      },
      "source": [
        "In addition to the dense vector movie index we have already created and used we now need to create a separate sparse index. We will query both indices and combine their recommendations to construct the final ranking for `user` + `text_query`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7gx_Sog0fHD"
      },
      "source": [
        "### Create an integrated sparse movie index"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AJJMdEMm_0gm"
      },
      "outputs": [],
      "source": [
        "# prepare sparse index\n",
        "sparse_index_name = \"sparse-movie-index\"\n",
        "\n",
        "if not pc.has_index(sparse_index_name):\n",
        "  pc.create_index_for_model(\n",
        "    name=sparse_index_name,\n",
        "    embed={\n",
        "        \"model\" : \"pinecone-sparse-english-v0\",\n",
        "        \"field_map\" : {\n",
        "            \"text\" : \"description\",\n",
        "        },\n",
        "    },\n",
        "    cloud=\"aws\",\n",
        "    region=\"us-east-1\"\n",
        "  )\n",
        "\n",
        "# Now connect to it\n",
        "sparse_index = pc.Index(sparse_index_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbGQEfhbsoDj"
      },
      "source": [
        "We did something differntly than before, well two things actually:\n",
        "1. We are working with a [`sparse_index`](https://docs.pinecone.io/guides/indexes/understanding-indexes). Such indecis allow us to perform **lexical** or **keyword search**\n",
        "2. We won't be embedding the text ourselves, we will rely on `Pinecone` to do this with us. This is why we used `pc.create_index_for_model()`, we already specified that `pinecone-sparse-english-v0` is the model used for embedding the inputs into this index.\n",
        "\n",
        "\n",
        "Let's take an example encoding:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2BEgWmButP1g",
        "outputId": "bb0e7194-a741-4feb-aed0-ecbf328170ba"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "EmbeddingsList(\n",
              "  model='pinecone-sparse-english-v0',\n",
              "  vector_type='sparse',\n",
              "  data=[\n",
              "    {'vector_type': sparse, 'sparse_values': [0.2849121, 0.34326172, ..., 1.9628906, 0.36767578], 'sparse_indices': [131900689, 152217691, ..., 4254144768, 4283091697], 'sparse_tokens': ['in', 'to', ..., 'follows', 's']}\n",
              "  ],\n",
              "  usage={'total_tokens': 58}\n",
              ")"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pc.inference.embed(\n",
        "    model=\"pinecone-sparse-english-v0\",\n",
        "    inputs=[\"The Toy Story franchise, starting with the 1995 film, follows the adventures of a group of toys who come to life when their owner, Andy, isn't around, with the story exploring themes of friendship, loyalty, and the changing roles of toys in a child's life\"],\n",
        "    parameters={\n",
        "    \"input_type\": \"passage\", # or query\n",
        "    \"return_tokens\": True,\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QweKw3Mjtu-q"
      },
      "source": [
        "### Batch upsert `text` into sparse movie index\n",
        "Let's upsert our movies into the index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K-kiI78rsIFk"
      },
      "outputs": [],
      "source": [
        "# transform movies into Pinecone sparse friendly form\n",
        "movie_embs_pc_sparse = pd.DataFrame({\n",
        "    'id': movie_embs['movieId'],\n",
        "    'description': movie_embs['title'] + \": \" + movie_embs['genres'].apply(lambda s: \" \".join(s.split(\"|\")),),  #+ \" \".join(movie_embs['genres'].split(\"|\")),\n",
        "})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NMs468HQv5sg"
      },
      "outputs": [],
      "source": [
        "# batching function\n",
        "def chunks(iterable, batch_size=32):\n",
        "    \"\"\"Break iterable into chunks of size batch_size.\"\"\"\n",
        "    it = iter(iterable)\n",
        "    chunk = tuple(itertools.islice(it, batch_size))\n",
        "    while chunk:\n",
        "        yield chunk\n",
        "        chunk = tuple(itertools.islice(it, batch_size))\n",
        "\n",
        "# data generator for batching\n",
        "def data_generator(df):\n",
        "    for _, row in df.iterrows():\n",
        "        yield (row['id'], row['description'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tPVFMTvcxJVF"
      },
      "outputs": [],
      "source": [
        "# iterate over batches and upsert into the sparse index\n",
        "for chunk_batch in chunks(data_generator(movie_embs_pc_sparse), batch_size=32):\n",
        "    text_to_upsert = [\n",
        "        {\n",
        "            'id': vec_id,\n",
        "            'description': desc,\n",
        "        }\n",
        "        for vec_id, desc in chunk_batch\n",
        "    ]\n",
        "    sparse_index.upsert_records(\n",
        "        namespace=\"\",\n",
        "        records=text_to_upsert\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7JIb9jSQEIp"
      },
      "source": [
        "We have successfully upserted the movie data into the sparse index. We can now use this index to perform hybrid search over both text and user similarity. Moreover, we didn't have to actually embedd anything ourselves, we used `pinecone-sparse-english-v0` hosted by Pinecone to do it instead!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "th6PkkesRGWz"
      },
      "source": [
        "### Perform Hybrid search\n",
        "We now have everything we need to develop our hybrid search application. For valuable references see:\n",
        "- [Query a sparse-dense index with explicit weighting](https://docs.pinecone.io/guides/indexes/pods/query-sparse-dense-vectors#query-a-sparse-dense-index-with-explicit-weighting)\n",
        "- [Introducing cascading retrieval: Unifying dense and sparse with reranking](https://www.pinecone.io/blog/cascading-retrieval/)\n",
        "\n",
        "In short we will, given the input of `userId` and `text_query`:\n",
        "1. retrieve many `movie` candidates with the `text_query` (using `pinecone-sparse-english-v0`)\n",
        "2. fetch the `user` embedding from the `users` index\n",
        "3. query the `movie` index and filter by metadata to get scores for all the retrieved `movieId`s\n",
        "4. re-rank with a simple re-ranking function based on both `sparse` and `dense` score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ccNaklsY6gOT"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "def rerank_candidates_with_dense(\n",
        "    sparse_candidates,\n",
        "    user_vector,\n",
        "    dense_movie_index,\n",
        "    alpha=0.5,\n",
        "    sort_score='hybrid_score'  # can be 'sparse_score' or 'dense_score'\n",
        "):\n",
        "    \"\"\"\n",
        "    1) Extract the list of movie IDs from sparse_candidates.\n",
        "    2) Query those items from 'dense_movie_index' using a filter.\n",
        "    3) Use the returned '.score' from Pinecone as 'dense_score'.\n",
        "    4) Return a new sorted list of dicts:\n",
        "        {\n",
        "          'movieId': ...,\n",
        "          'sparse_score': ...,\n",
        "          'dense_score': ...,\n",
        "          'hybrid_score': ...,\n",
        "          'metadata': ...\n",
        "        }\n",
        "       sorted descending by 'hybrid_score'. Where:\n",
        "       - 'hybrid_score' = alpha * 'sparse_score' + (1 - alpha) * 'dense_score'\n",
        "    \"\"\"\n",
        "\n",
        "    if not user_vector:\n",
        "        return []\n",
        "\n",
        "    # 1) Gather movie IDs\n",
        "    movie_ids = [r['_id'] for r in sparse_candidates]\n",
        "\n",
        "    # 2) Filtered query to Pinecone.\n",
        "    query_res = dense_movie_index.query(\n",
        "        vector=user_vector,\n",
        "        filter={\"movieId\": {\"$in\": movie_ids}},\n",
        "        top_k=len(movie_ids),\n",
        "        include_values=False,\n",
        "        include_metadata=True\n",
        "    )\n",
        "\n",
        "    # Build a lookup {id: match_object}\n",
        "    dense_matches = {}\n",
        "    if query_res and query_res.matches:\n",
        "        dense_matches = {m.id: m for m in query_res.matches}\n",
        "\n",
        "    # 3) Combine sparse_score + dense_score\n",
        "    results = []\n",
        "    for item in sparse_candidates:\n",
        "        rid = str(item['_id'])\n",
        "        sparse_score = item.get('_score', 0.0)\n",
        "        match_data = dense_matches[rid]\n",
        "        dense_score = match_data.score\n",
        "\n",
        "        # Weighted hybrid score\n",
        "        hybrid_score = alpha * sparse_score + (1 - alpha) * dense_score\n",
        "\n",
        "        results.append({\n",
        "            'movieId': rid,\n",
        "            'sparse_score': sparse_score,\n",
        "            'dense_score': dense_score,\n",
        "            'hybrid_score': hybrid_score,\n",
        "            'metadata': match_data.metadata or {}\n",
        "        })\n",
        "\n",
        "    # 4) Sort descending by 'sort_score'\n",
        "    results.sort(key=lambda x: x[sort_score], reverse=True)\n",
        "    return results\n",
        "\n",
        "\n",
        "def retrieve_and_rerank(\n",
        "    user_id,\n",
        "    text_query,\n",
        "    user_index,\n",
        "    sparse_index,\n",
        "    dense_index,\n",
        "    top_k_sparse=20,\n",
        "    top_k_final=10,\n",
        "    alpha=0.5\n",
        "):\n",
        "    # 1) Sparse retrieval\n",
        "    sparse_candidates = sparse_index.search_records(\n",
        "        namespace=\"\",\n",
        "        query={\n",
        "            \"inputs\": {\"text\": text_query},\n",
        "            \"top_k\": top_k_sparse,\n",
        "        },\n",
        "        fields=[\"description\"]\n",
        "    ).result.hits\n",
        "    if not sparse_candidates:\n",
        "        print(\"No sparse candidates found.\")\n",
        "        return []\n",
        "\n",
        "    # 2) Get user's dense embedding\n",
        "    user_vec = user_index.fetch(ids=[user_id]).vectors[user_id].values\n",
        "    if user_vec is None:\n",
        "        print(f\"No embedding found for user '{user_id}'. Cannot re-rank.\")\n",
        "        return sparse_candidates  # or return []\n",
        "\n",
        "    # 3) Re-rank by using Pinecone's internal similarity score\n",
        "    final_results = rerank_candidates_with_dense(\n",
        "        sparse_candidates,\n",
        "        user_vec,\n",
        "        dense_index,\n",
        "        alpha=alpha\n",
        "    )\n",
        "\n",
        "    return final_results[:top_k_final]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g7Xv6cIjR4mp"
      },
      "source": [
        "Function to putting it all together:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bz9O0r-9B0gR"
      },
      "outputs": [],
      "source": [
        "def display_hybrid_results(matches, heading=\"Hybrid Search Results\", sort_score='hybrid_score'):\n",
        "    html_str = f\"<h4>{heading}</h4><div style='display:flex;flex-wrap:wrap;'>\"\n",
        "    if not matches:\n",
        "        html_str += \"<p>No results found.</p>\"\n",
        "    else:\n",
        "        for match in matches:\n",
        "            md = match.get('metadata') or {}\n",
        "            title = md.get('title', 'Unknown Title')\n",
        "            poster_url = md.get('poster', '')\n",
        "            genres = md.get('genres', '')\n",
        "            score_val = match.get(sort_score)\n",
        "\n",
        "            html_str += f\"\"\"\n",
        "            <div style='margin:10px;text-align:center;'>\n",
        "                <img src=\"{poster_url}\" alt=\"{title}\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
        "                <div>{title}</div>\n",
        "                <div>Genres: {genres}</div>\n",
        "                <div>Score: {score_val:.4f}</div>\n",
        "            </div>\n",
        "            \"\"\"\n",
        "    html_str += \"</div>\"\n",
        "    display(HTML(html_str))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ExN8F_FR-HX"
      },
      "source": [
        "And it's as easy as that, feel free to play with the `user` anchor and the `text_query` below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "YkvSlkRvawi9",
        "outputId": "b709c607-0be0-4cf9-dbdc-7890644521ea"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"user_embs\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"userId\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"5839\",\n          \"83399\",\n          \"47698\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"EMBEDDING\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-7a456cab-b4f0-4bb3-8b37-16e7053304f5\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "      <th>EMBEDDING</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4383</th>\n",
              "      <td>56180</td>\n",
              "      <td>[0.44935083, 0.18861422, -0.40817547, 0.574801...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26134</th>\n",
              "      <td>5839</td>\n",
              "      <td>[0.16136876, 0.056439295, -0.49999896, 0.16478...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8658</th>\n",
              "      <td>47698</td>\n",
              "      <td>[-0.011269495, 0.0056284517, -0.46137083, 0.01...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16262</th>\n",
              "      <td>73940</td>\n",
              "      <td>[0.33393583, 0.2415386, -0.51072073, 0.3245950...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9023</th>\n",
              "      <td>83399</td>\n",
              "      <td>[0.35129237, 0.25972885, -0.36625588, 0.224390...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7a456cab-b4f0-4bb3-8b37-16e7053304f5')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7a456cab-b4f0-4bb3-8b37-16e7053304f5 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7a456cab-b4f0-4bb3-8b37-16e7053304f5');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0cbbdfbb-6a0e-4ba4-9519-7c59ba9e952a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0cbbdfbb-6a0e-4ba4-9519-7c59ba9e952a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0cbbdfbb-6a0e-4ba4-9519-7c59ba9e952a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "      userId                                          EMBEDDING\n",
              "4383   56180  [0.44935083, 0.18861422, -0.40817547, 0.574801...\n",
              "26134   5839  [0.16136876, 0.056439295, -0.49999896, 0.16478...\n",
              "8658   47698  [-0.011269495, 0.0056284517, -0.46137083, 0.01...\n",
              "16262  73940  [0.33393583, 0.2415386, -0.51072073, 0.3245950...\n",
              "9023   83399  [0.35129237, 0.25972885, -0.36625588, 0.224390..."
            ]
          },
          "execution_count": 81,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "user_embs.sample(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 579
        },
        "id": "0SpIx5ZqB3vg",
        "outputId": "43bcf106-f6e7-460b-befa-cc6ebbc945a3"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<h4>Cascading/Hybrid Search for 'Sci-fi'</h4><div style='display:flex;flex-wrap:wrap;'>\n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMTExMzU0ODcxNDheQTJeQWpwZ15BbWU4MDE1OTI4MzAy._V1_SX300.jpg\" alt=\"Arrival (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Arrival (2016)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 5.4448</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BNjk2OTc5YzQtMjAwMS00YWY4LTk1ZWItOTgyMmRkMGU4ZmY1XkEyXkFqcGdeQXVyMzQ1MzUwMTE@._V1_SX300.jpg\" alt=\"Allegiant: Part 1 (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Allegiant: Part 1 (2016)</div>\n",
              "                <div>Genres: Adventure|Sci-Fi</div>\n",
              "                <div>Score: 4.3081</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMzhmMWY5YzYtNGU0OS00OWExLWE4MTEtZjdmZTczNGEwNjhmXkEyXkFqcGdeQXVyMjM4NTM5NDY@._V1_SX300.jpg\" alt=\"A.X.L. (2018)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>A.X.L. (2018)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 4.3063</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMTUxMjQ2NjI4OV5BMl5BanBnXkFtZTgwODc2NjUwNDE@._V1_SX300.jpg\" alt=\"Project Almanac (2015)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Project Almanac (2015)</div>\n",
              "                <div>Genres: Sci-Fi|Thriller</div>\n",
              "                <div>Score: 4.1745</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BNmU4NTc0ZTgtNjliOC00NTM2LWE3NDktNGJiNzc2YzY3ZjA2XkEyXkFqcGdeQXVyNDg4NjY5OTQ@._V1_SX300.jpg\" alt=\"IO (2019)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>IO (2019)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 4.0488</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMjAxODQ2MzkyMV5BMl5BanBnXkFtZTgwNjU3MTE5OTE@._V1_SX300.jpg\" alt=\"ARQ (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>ARQ (2016)</div>\n",
              "                <div>Genres: Sci-Fi|Thriller</div>\n",
              "                <div>Score: 4.0263</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMTU5OTYzMzcwOF5BMl5BanBnXkFtZTgwNTkzMzk4NTM@._V1_SX300.jpg\" alt=\"Extinction (2018)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Extinction (2018)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.8757</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BODNkNTFlMWItZWNkYS00YTUwLWJkN2YtYWQxMGJlZDA0OGFlXkEyXkFqcGdeQXVyMjcxNjI4NTk@._V1_SX300.jpg\" alt=\"Attraction (2017)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Attraction (2017)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.8161</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BY2YyMDM0OTktNTA4MC00MDBjLTg5NTAtMzMxZTAwNmNjMzllXkEyXkFqcGdeQXVyNjM4OTY2Njc@._V1_SX300.jpg\" alt=\"\u041c\u0430\u0442\u0435\u043c\u0430\u0442\u0438\u043a \u0438 \u0447\u0451\u0440\u0442 (1972)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>\u041c\u0430\u0442\u0435\u043c\u0430\u0442\u0438\u043a \u0438 \u0447\u0451\u0440\u0442 (1972)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.7455</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BNmYzYWNhYWQtNjg0MS00NWU5LTgyZGUtYzViNjgwY2NjY2YwXkEyXkFqcGdeQXVyNjYzMzU4OTg@._V1_SX300.jpg\" alt=\"Tau\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Tau</div>\n",
              "                <div>Genres: Sci-Fi|Thriller</div>\n",
              "                <div>Score: 3.6148</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMjAwNzA4OTE0Ml5BMl5BanBnXkFtZTgwODg3NTczMzI@._V1_SX300.jpg\" alt=\"Realive (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Realive (2016)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.6015</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BMjE4NDc5MDU2NV5BMl5BanBnXkFtZTgwNDI1ODEyNTM@._V1_SX300.jpg\" alt=\"The Titan (2018)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>The Titan (2018)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.5879</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BZmI0MjhjNjItZmU2MC00NmI1LTllOGQtODUyNGNmYjIyNmJmXkEyXkFqcGdeQXVyMTY2NTQ3ODc@._V1_SX300.jpg\" alt=\"2036 Origin Unknown (2018)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>2036 Origin Unknown (2018)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.5846</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BYjc0MGQ1ODMtNjE5MS00MzU5LTg5MzgtNzQxNmY0NjU3YzViXkEyXkFqcGdeQXVyMjcxNjI4NTk@._V1_SX300.jpg\" alt=\"Calculator (2014)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Calculator (2014)</div>\n",
              "                <div>Genres: Sci-Fi</div>\n",
              "                <div>Score: 3.5392</div>\n",
              "            </div>\n",
              "            \n",
              "            <div style='margin:10px;text-align:center;'>\n",
              "                <img src=\"https://m.media-amazon.com/images/M/MV5BYzNmNTA3NzktYTAzZC00ZmZiLWIwOWMtODMxYmMwMDE3MTBhXkEyXkFqcGdeQXVyMjExNjgyMTc@._V1_SX300.jpg\" alt=\"Code 8 (2016)\" width=\"120\" style='display:block;margin-bottom:5px;'/>\n",
              "                <div>Code 8 (2016)</div>\n",
              "                <div>Genres: Action|Sci-Fi</div>\n",
              "                <div>Score: 3.4669</div>\n",
              "            </div>\n",
              "            </div>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "user_id_example = \"83973\"\n",
        "text_query = \"Sci-fi\"\n",
        "\n",
        "final_recs = retrieve_and_rerank(\n",
        "    user_id=user_id_example,\n",
        "    text_query=text_query,\n",
        "    user_index=user_index,\n",
        "    sparse_index=sparse_index,\n",
        "    dense_index=movie_index,\n",
        "    top_k_sparse=100,\n",
        "    top_k_final=15,\n",
        "    alpha=0.7,\n",
        ")\n",
        "\n",
        "display_hybrid_results(\n",
        "    final_recs,\n",
        "    heading=f\"Cascading/Hybrid Search for '{text_query}'\",)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kcikylzg3EtE"
      },
      "source": [
        "# Conclusion\n",
        "In this notebook example we went on a journey of building our own `user` and `movie` embeddings with Kumo and building out actual personalization solutions in Pinecone.\n",
        "\n",
        "The recipes given in this notebook are meant to be very general so we are looking forward to seeing both Kumo and Pinecone applied in may exciting applications.\n",
        "\n",
        "For any questions feel free to consult both Kumo or Pinecone docs:\n",
        "- [Kumo docs](https://kumo.ai/docs/overview)\n",
        "- [Pinecone docs](https://docs.pinecone.io/guides/get-started/overview)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}